r/SEO

10h ago

u/Novel_Buy_7171

Increase in 429 errors during crawls?

Hey all, over the last couple of weeks I've seen a big increase in the number of 429nerrors appearing in crawls, even with rate limits, this is across tools such as screaming frog and SEM rush.

I was wondering if anyone else has been seeing this?

Pet theory right now is that servers are being way more strict on crawls with the rise of ai crawling (my first thought was this might be cloud flare, but similar issues are happening on sites without cloud flare as well).

Roblox

•

Ad

Sound on. Lights off. Discover survival games on Roblox.

Play Now roblox.com

Best

Open comment sort options

Comments Section

WxMan2201

8h ago

Top 1% Commenter

Yes - means that the crawler has been blocked by the servers for the website - probably too many requests at once, during a certain period.…