Menu

Next.js prefetches got me rate-limited on my own site. Googlebot had been fetching them by the thousand
📰

Next.js prefetches got me rate-limited on my own site. Googlebot had been fetching them by the thousand

DEV Community: react·Alex Ialanji·5 days ago
#WxG0fbVx
#devtoreact#devto#dev#_rsc#googlebot
Reading 0:00
15s threshold

This article was created with the help of AI. The numbers come from my own site's Search Console and server logs.

On October 8 I opened my own news site, promtime.net, on my phone and got Cloudflare's error 1015 instead of a page. My own rate-limiting rule had blocked me, and all I had done was read posts on it.

The cause was a Next.js feature that exists to make the site feel fast. Two days later Search Console showed me the same feature from Googlebot's side. Between August 10 and October 22 Googlebot made 25,943 requests to the site, and Search Console filed 16,284 of them under "Other file type": not pages, not images, not scripts.

What the router does

When a <Link> scrolls into view, the Next.js router prefetches the page behind it. It requests the same URL with an _rsc parameter and an RSC: 1 header and gets back the React Server Components payload instead of HTML, so the click that follows feels instant.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.