Hey everyone! 👋

Over the last few months, I noticed a huge spike in aggressive web crawlers like Bytespider , GPTBot , and ClaudeBot hitting web servers. Unlike Google or Bing, many of these AI training scrapers ignore standard crawl delays and scrape content without sending any referral traffic back.

Manually looking up user-agents and writing complex rules for both robots.txt and reverse proxies was getting tedious, so I built CrawlGuard .

👉 Try CrawlGuard Free at crawlguard.dev

What it does:

1-Click Presets: Generate optimal rules (e.g., "Block All AI Training", "E-Commerce Protection", or "Balanced Modern Web").

Multi-Format Export: Spits out valid RFC 9309 robots.txt , Apache .htaccess , Nginx .conf , and Cloudflare WAF rules for bots that ignore robots directives.

Live Tester: Includes an interactive simulator at the bottom to test if your URL paths are allowed or blocked.

100% Free & Private: Runs client-side in your browser with no login or account required.

I’d love to hear your feedback…