Menu

How to check if robots.txt is blocking Google (and fix it without breaking anything)
📰

How to check if robots.txt is blocking Google (and fix it without breaking anything)

DEV Community: beginners·seotto.com·17 days ago
#BqHzPHk5
#devtobeginners#devto#dev#robots#google
Reading 0:00
15s threshold

robots.txt is a plain text file with a lot of power. One line in it can stop Google from crawling your entire site, and because every page still opens normally in a browser, nobody notices until traffic falls. Here is how to check whether yours is blocking Google, and how to read the rules correctly.

Step 1: Find the file

robots.txt always lives at the root of the host:

https://example.com/robots.txt

Enter fullscreen mode

Exit fullscreen mode

Note that each host has its own: www.example.com , example.com , blog.example.com and the http:// versions can all serve different files.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.