Python Web Scraper: Extract Data from Any Website in 50 Lines

Web scraping is one of the most practical Python skills. Here's a production-ready scraper that handles most websites.

Setup

pip install requests beautifulsoup4 lxml

Enter fullscreen mode

Exit fullscreen mode

Core Scraper

import requests from bs4 import BeautifulSoup import json import time

def scrape_page ( url : str ) -> BeautifulSoup : headers = { ' User-Agent ' : ' Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' } response = requests . get ( url , headers = headers , timeout = 15 ) response . raise_for_status () return BeautifulSoup ( response . text , ' lxml ' )

# Example: Scrape Hacker News front page def scrape_hackernews (): soup = scrape_page ( ' https://news.ycombinator.com ' ) stories = []

for item in soup . select ( ' .athing ' ): title_el = item . select_one ( ' .titleline > a ' ) if title_el : stories . append ({ ' title ' : title_el . text , ' url ' : title_el .…