I got tired of maintaining Puppeteer scripts and proxy lists for every website I needed data from. So I built something better -- a single API that handles JS rendering, proxy rotation, CAPTCHA solving, and AI-powered extraction.
How It Works
1. Scrape any page
\\ash
curl -X POST ' https://run.xcrawl.com/v1/scrape ' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{"url": " https://news.ycombinator.com ", "output": {"formats": ["markdown"]}}'
\\
Returns clean Markdown. No HTML parsing, no DOM traversal.
2. AI extraction
Describe what you want in English, get structured JSON:
\\ash
curl -X POST ' https://run.xcrawl.com/v1/scrape ' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{"url": " https://example.com ", "output": {"formats": ["json"]}, "json": {"prompt": "Extract product names and prices"}}'
\\
3.…