Sign-in pages are the classic thing to hide from search. The reflex is to put them in robots.txt and move on. On RadarMakers we did exactly that, and then changed our minds about one path. This post shows the two small edits, the reasoning behind them, and, just as important, what the change does not prove.

The rule that is easy to get backwards

robots.txt and noindex answer different questions. robots.txt says whether a crawler may fetch a URL. noindex says whether a page that was fetched may appear in search results. The second one can only be read from a response, so a crawler that is not allowed to fetch the URL never sees it.

Google's documentation on blocking indexing with noindex says this directly: for a noindex rule to be effective, the page must not be blocked by robots.txt .…