Most AI study path generators fail in a predictable way: they hallucinate dead links, ignore real-world weekly time constraints, or link to "free" courses that suddenly demand a credit card on Week 3.
I built Episteme AI ( live demo ) to solve this at the pipeline level. Here is how the paywall detection and constraint enforcement work under the hood.
1. Why HTTP 200 OK Isn't Enough
Many course pages return HTTP 200 while locking content behind a paywall, subscription, or 7-day trial.
Rather than trusting status codes, we fetch the target page using rotating browser headers and inspect the DOM directly with Cheerio:
// From scripts/check-url.mjs & src/lib/paywallInspector.mjs const $ = cheerio . load ( html ) ;
// 1. Check Schema.org JSON-LD for explicit open access markers let isAccessibleForFree = false ; $( 'script[type="application/ld+json"]' ) . each ( ( _ , el ) => { try { const data = JSON . parse ( $( el ) . html ( ) ) ; if ( data . isAccessibleForFree === true || data . offers ? .…