robots.txt unreachable is not usually a content problem.
It is usually a fetch, routing, DNS, CDN, middleware, firewall, redirect, or cache problem.
That distinction matters because teams often waste time editing pages when Google is really saying:
"I could not reliably fetch the file that tells me what I am allowed to crawl."
Here is the debugging order I use.
1. Confirm the file exists at the root
Open:
https://example.com/robots.txt
Enter fullscreen mode
Exit fullscreen mode
It should return a plain-text response from the same public host Google crawls.
2. Check the HTTP status
Use:
curl -I https://example.com/robots.txt
Enter fullscreen mode
Exit fullscreen mode
You want a stable 200 OK .
Watch for:
403 from bot protection
404 from routing
5xx from hosting or edge functions
long redirect chains
HTML being returned instead of plain text
3.…