Menu

Turn any web page into typed JSON with an LLM and a JSON Schema (Python)
📰

Turn any web page into typed JSON with an LLM and a JSON Schema (Python)

DEV Community: python·William Jin·20 days ago
#oR0fmAHN
#devtopython#devto#dev#books#currency
Reading 0:00
15s threshold

Writing CSS selectors for every site you want data from gets old fast, and the selectors break when the site changes its markup. For many jobs (a product list, an events page, a team page) it's simpler to give an LLM the page text and a schema and let it fill in the fields.

The naive version (paste the HTML, ask for JSON) works in a demo and fails in production: the output shape drifts, links vanish, and long pages blow through the context window. This post shows a version that holds up, using a real page you can try: the Travel category of books.toscrape.com , a sandbox site built for scraping practice.

The pattern

Fetch and clean. Strip scripts, styles, navigation and footers. They cost tokens and add noise.

Keep the link targets. After you convert to text, the model can't return a URL it never saw, so write text (url) in place of each link.

Cap the size. Trim to a fixed budget before sending.

Force the shape.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.