Python Web Scraping: Extract Data from Any Website in 2024

Web scraping is one of the most useful Python skills. Here is how to do it right.

Basic Scraping with BeautifulSoup

import requests from bs4 import BeautifulSoup

def scrape_page ( url ): headers = { " User-Agent " : " Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " } response = requests . get ( url , headers = headers , timeout = 10 ) soup = BeautifulSoup ( response . text , " html.parser " ) return soup

Enter fullscreen mode

Exit fullscreen mode

Extract Common Data

# Get all links links = [ a [ ' href ' ] for a in soup . find_all ( ' a ' , href = True )]

# Get text by CSS selector titles = [ el . text . strip () for el in soup . select ( ' h2.article-title ' )]

# Get table data rows = [] for tr in soup . select ( ' table tr ' ): row = [ td . text . strip () for td in tr . find_all ( ' td ' )] if row : rows .…