Googlers Gary Illyes and Cherry Prommawin explain how moving to a CDN can unexpectedly increase crawling and overwhelm the origin server, despite eventually improving website performance. They also share that attempts to control crawler traffic can sometimes accidentally block Googlebot and harm Search visibility.
Moving To A CDN Can Cause Googlebot To Increase Crawling
Illyes and Prommawin explained that moving to a CDN can cause Googlebot to increase crawling because faster server responses signal that the site’s infrastructure can handle more requests. Google’s automated crawling systems interpret the improved performance as an indication that more crawling is possible and responds by gradually increasing the crawl rate.
CDN caches are initially empty and the CDN only starts caching when a browser or bot requests a page at which point the request has to be forwarded to the origin server, where the actual website is at.
When Google increases the crawl rate, the initial crawl is handled by the origin…