Menu

Googlebot Saw Empty Tables on Our JavaScript Site: What We Found in Search Console
πŸ“°

Googlebot Saw Empty Tables on Our JavaScript Site: What We Found in Search Console

DEV Community: javascriptΒ·vunelixΒ·18 days ago
#5cy3sWDk
#devtojavascript#devto#dev#googlebot#google
Reading 0:00
15s threshold

Our site is a set of market-data tools: screeners, heatmaps and calculators. Tables are filled by JavaScript from a small backend endpoint. For months Search Console showed "Crawled - currently not indexed" for almost every tool page, and we blamed content quality. Then we looked at two reports we had never opened, and found two plain technical bugs.

This post is from the Vunelix team and the site in question is ours. The lessons apply to any JavaScript-rendered site.

Bug 1: 96% of Googlebot's requests went to one endpoint that always failed

In Search Console, open Settings β†’ Crawl stats β†’ By response . Ours showed about 7.8 million Googlebot requests in 90 days, and 96% of them were "Other client error (4XX)". The example URLs were all the same path: a POST-only endpoint that our embeddable widgets call.

The reason: our widgets run on other people's sites. Googlebot renders those pages too, runs the widget script, and posts to our endpoint about 80,000 times a day.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.