My site answered 200 on all 163 URLs. Crawlers never saw a single page.

For two months I ran a small data site: 163 URLs, 152 generated company pages, a 1.7 MB CSV, valid sitemap.xml , and an IndexNow submission that returned 200 every single week.

Every check I had was green. And every search engine had zero of my pages.

This is the story of what was wrong, because I think it is a common failure and the symptom looks exactly like success.

The clue was one request away

On a hunch I fetched the same file twice:

curl -s https://example.site/robots.txt | wc -c # 26

curl -s "https://example.site/robots.txt?probe=1" | wc -c # 162

Enter fullscreen mode

Exit fullscreen mode

Two different files from the same path.…