Gary Illyes and Chery Prommawin released their second video in the Non-Canonical Cabinet series; this one is named What can you actually put in a robots.txt? Here they discuss what is, and is not, supported by robots.txt files. And while Google may support features or directives, other search engines or AI crawlers might not.
Here are the top level takeaways:
Googlebot uses the robots.txt file to limit crawling
Your robot.txt file should follow RFC 9309
Use percent-encoding for non-English characters in rule paths
And check out Google's document for more details
Here are some points I pulled out from listening to the video:
Don't assume a robots.txt rule is working just because it looks understandable to a human. Validate the syntax and test the actual URL patterns.
Be careful when blocking URLs containing international characters or emojis. Encoding inconsistencies can undermine a rule, particularly across different crawler implementations.
Separate crawling controls from AI training policies.…