I ran one query against my AI tools directory last night, and the result explained why some of its pages get crawled while others sit ignored.
Category Tools
RIP AI 405
Video Generators 294
Image Generators 269
Productivity 253
Developer Tools 247
Education / Studies 245
Business 233
Image editing 230
LLM models 208
Life Assistants 192
Those ten categories hold 2,576 of 7,006 category assignments β 36.8% of the whole taxonomy. The directory has 4,323 tools across 92 categories.
That last number is not the problem. The tail is.
14 categories have ten tools or fewer
New AI Tools: 0
AIxploria Selection: 0
Programming: 0
AI Presentations: 2
AI Music: 4
AI Education: 4
AI 3D: 4
AI Research: 4
AI SEO: 5
AI Translation: 7
AI Chatbots: 8
AI Image Generators: 8
AI Social Media: 8
AI Customer Support: 10
Three of those are empty, and they are also the most tempting pages to publish, because the names sound valuable: "New AI Tools" and "Programming" are exactly what a searcher types.β¦