Our site is a set of market-data tools: screeners, heatmaps and calculators. Tables are filled by JavaScript from a small backend endpoint. For months Search Console showed "Crawled - currently not indexed" for almost every tool page, and we blamed content quality. Then we looked at two reports we had never opened, and found two plain technical bugs.
This post is from the Vunelix team and the site in question is ours. The lessons apply to any JavaScript-rendered site.
Bug 1: 96% of Googlebot's requests went to one endpoint that always failed
In Search Console, open Settings β Crawl stats β By response . Ours showed about 7.8 million Googlebot requests in 90 days, and 96% of them were "Other client error (4XX)". The example URLs were all the same path: a POST-only endpoint that our embeddable widgets call.
The reason: our widgets run on other people's sites. Googlebot renders those pages too, runs the widget script, and posts to our endpoint about 80,000 times a day.β¦