Menu

robots.txt for a downloader: what I allow search engines to crawl and what I block
πŸ“°

robots.txt for a downloader: what I allow search engines to crawl and what I block

DEV Community: phpΒ·Mobeen AhmedΒ·about 10 hours ago
#t4DvTezf
#devto#dev#pages#file#robots
Long article (684 words) β€” no AI summary yet.
Reading 0:00
15s threshold

When I built ramrom.io, a small video downloader that runs on plain PHP hosting, I spent most of my time on the tool itself: pasting a link, resolving it, handing the visitor a file. The robots.txt file was an afterthought, a one-line file I copied from somewhere and forgot about. That was a mistake, and it took me a while to understand why it matters for a site like this.

A downloader site is a strange thing for a search engine to crawl. Most of the site is a handful of real pages: the homepage, a page for each platform, a few guides, a blog. But behind those pages sit the endpoints that do the actual work: the resolve endpoint that talks to the provider APIs, and the download proxy that streams the file back. Those are not pages. There is nothing for a search engine to index there, and nothing a visitor would ever want to find in search results.

The problem is that a crawler does not know that. If it finds a link to an API endpoint, or guesses a path, it will happily request it.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.