Every few months someone on a dev team asks me, "Should we be worried about crawl budget?" The site in question usually has 200 pages and runs on a $10 VPS. The honest answer is almost always "no, not directly." But that answer hides something useful: the bugs that would burn crawl budget on a big site still exist on small ones, and they usually show up as indexing problems you're already confused about.

This post is the developer/ops version. Less theory, more logs, curl , and config files.

The 30-second definition

Googlebot decides how many URLs to fetch from your host based on two things:

Capacity: how fast and reliably your server responds. Slow responses or 5xx errors make it back off.

Demand: how valuable and fresh it thinks your URLs are.

If your site has a few hundred real URLs and responds in under a second, Google can crawl everything many times over. Budget isn't your bottleneck.…