Menu

1 / 2

Google On robots.txt Rules For AI Crawlers, Emojis, International URLs & More

Search Engine Roundtable·Barry Schwartz·25 days ago
#d9FLC2wy
#gary#illyes#cherry#prommawin#chicken
Reading 0:00
15s threshold

Gary Illyes and Chery Prommawin released their second video in the Non-Canonical Cabinet series; this one is named What can you actually put in a robots.txt? Here they discuss what is, and is not, supported by robots.txt files. And while Google may support features or directives, other search engines or AI crawlers might not.

Here are the top level takeaways:

Googlebot uses the robots.txt file to limit crawling

Your robot.txt file should follow RFC 9309

Use percent-encoding for non-English characters in rule paths

And check out Google's document for more details

Here are some points I pulled out from listening to the video:

Don't assume a robots.txt rule is working just because it looks understandable to a human. Validate the syntax and test the actual URL patterns.

Be careful when blocking URLs containing international characters or emojis. Encoding inconsistencies can undermine a rule, particularly across different crawler implementations.

Separate crawling controls from AI training policies.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.