Menu

1 / 2

I Built a Web Scraper API That Handles JS Rendering, CAPTCHAs, and Proxies

DEV Community·Charles·6 months ago
#zUjoT9uw
Reading 0:00
15s threshold

I got tired of maintaining Puppeteer scripts and proxy lists for every website I needed data from. So I built something better -- a single API that handles JS rendering, proxy rotation, CAPTCHA solving, and AI-powered extraction.

How It Works

1. Scrape any page

\\ash

curl -X POST ' https://run.xcrawl.com/v1/scrape ' \

-H 'Authorization: Bearer ***' \

-H 'Content-Type: application/json' \

-d '{"url": " https://news.ycombinator.com ", "output": {"formats": ["markdown"]}}'

\\

Returns clean Markdown. No HTML parsing, no DOM traversal.

2. AI extraction

Describe what you want in English, get structured JSON:

\\ash

curl -X POST ' https://run.xcrawl.com/v1/scrape ' \

-H 'Authorization: Bearer ***' \

-H 'Content-Type: application/json' \

-d '{"url": " https://example.com ", "output": {"formats": ["json"]}, "json": {"prompt": "Extract product names and prices"}}'

\\

3.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.