A page can return 200 OK, exist in an XML sitemap, and still be practically invisible to anyone browsing the site.
That is the orphan-page problem.
You do not need a full commercial crawler to find the obvious cases. A small Node.js script can compare two things:
URLs declared in the sitemap
and
URLs reachable through HTML links
Anything present in the first set but missing from the second deserves investigation.
Crawl the Link Graph
Install Cheerio and an XML parser:
npm install cheerio fast - xml - parser
Then create find-orphans.mjs:
import * as cheerio from "cheerio" ; import { XMLParser } from "fast-xml-parser" ; const startUrl = process . argv [ 2 ] ; const sitemapUrl = process . argv [ 3 ] ; const parser = new XMLParser ( ) ; function normalize ( input , base ) { try { const url = new URL ( input , base ) ; url . hash = "" ; return url . href ; } catch { return null ; } } async function crawl ( start ) { const origin = new URL ( start ) .…