Menu

Crawlee proxy setup: rotation, sessions, tiers and SOCKS5
📰

Crawlee proxy setup: rotation, sessions, tiers and SOCKS5

DEV Community: tutorial·proxymint·1 day ago
#XKQw6Scv
#devtotutorial#devto#dev#proxy#crawlee
Reading 0:00
15s threshold

Originally published on proxymint .

Crawlee proxy setup for Node and Python: ProxyConfiguration, how sessions pin proxies, the browser trap where proxyInfo lies, tiers, SOCKS5 and 407 loops.

TL;DR

ProxyConfiguration is the whole setup. Pass proxyUrls (Node) or proxy_urls (Python) with the login inside each URL, hand the object to the crawler, and read proxyInfo in the handler to see which URL Crawlee picked.

Rotation follows sessions. Each new session takes the next URL in round robin and keeps it. The default pool holds 1,000 sessions, so early in a crawl nearly every request gets a new session and the next proxy.

Browser crawlers can report the wrong proxy. In crawlee 3.18.2 a default PlaywrightCrawler logged three proxies in proxyInfo while every page exited through the first; useIncognitoPages or browserPerProxy fixed it.

socks5:// depends on the client. got-scraping, the Node default, accepts only HTTP proxy URLs, and Crawlee for Python rejects SOCKS URLs at construction.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.