Menu

How I Built a Self-Healing Study Planner: Catching Paywalled "Free" Courses with Cheerio and Schema.org
📰

How I Built a Self-Healing Study Planner: Catching Paywalled "Free" Courses with Cheerio and Schema.org

DEV Community: nextjs·Sree Naresh A·4 days ago
#jtt0mb3r
#devtonextjs#devto#dev#week#paywall
Reading 0:00
15s threshold

Most AI study path generators fail in a predictable way: they hallucinate dead links, ignore real-world weekly time constraints, or link to "free" courses that suddenly demand a credit card on Week 3.

I built Episteme AI ( live demo ) to solve this at the pipeline level. Here is how the paywall detection and constraint enforcement work under the hood.

1. Why HTTP 200 OK Isn't Enough

Many course pages return HTTP 200 while locking content behind a paywall, subscription, or 7-day trial.

Rather than trusting status codes, we fetch the target page using rotating browser headers and inspect the DOM directly with Cheerio:

// From scripts/check-url.mjs & src/lib/paywallInspector.mjs const $ = cheerio . load ( html ) ;

// 1. Check Schema.org JSON-LD for explicit open access markers let isAccessibleForFree = false ; $( 'script[type="application/ld+json"]' ) . each ( ( _ , el ) => { try { const data = JSON . parse ( $( el ) . html ( ) ) ; if ( data . isAccessibleForFree === true || data . offers ? .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.