You cannot make paid content unscrapable. You can make sure the bytes never leave your server unless someone paid, and you can make scraping expensive and pointless. This post is about where that line actually sits, and what a workable setup looks like.

What robots.txt actually does

robots.txt is a file of requests at the root of a domain. A crawler that chooses to follow it reads it before fetching anything else and skips the paths you list.

Three properties matter:

It is advisory. Nothing enforces it. A crawler that ignores it gets whatever your server hands out.

It is per-user-agent and prefix-matched. Disallow: /premium/ covers /premium/a and /premium/b/c , but not /articles/premium-a .

It is public.…