When I built ramrom.io, a small video downloader that runs on plain PHP hosting, I spent most of my time on the tool itself: pasting a link, resolving it, handing the visitor a file. The robots.txt file was an afterthought, a one-line file I copied from somewhere and forgot about. That was a mistake, and it took me a while to understand why it matters for a site like this.
A downloader site is a strange thing for a search engine to crawl. Most of the site is a handful of real pages: the homepage, a page for each platform, a few guides, a blog. But behind those pages sit the endpoints that do the actual work: the resolve endpoint that talks to the provider APIs, and the download proxy that streams the file back. Those are not pages. There is nothing for a search engine to index there, and nothing a visitor would ever want to find in search results.
The problem is that a crawler does not know that. If it finds a link to an API endpoint, or guesses a path, it will happily request it.β¦