Menu

1 / 3

Find Orphan Pages by Comparing Your Sitemap with an Internal Crawl

SitePoint·SitePoint Team·about 18 hours ago
#s1LxHaVM
#sitepoint#const#return#sitemap#article
Reading 0:00
15s threshold

A page can return 200 OK, exist in an XML sitemap, and still be practically invisible to anyone browsing the site.

That is the orphan-page problem.

You do not need a full commercial crawler to find the obvious cases. A small Node.js script can compare two things:

URLs declared in the sitemap

and

URLs reachable through HTML links

Anything present in the first set but missing from the second deserves investigation.

Crawl the Link Graph

Install Cheerio and an XML parser:

npm install cheerio fast - xml - parser

Then create find-orphans.mjs:

import * as cheerio from "cheerio" ; import { XMLParser } from "fast-xml-parser" ; const startUrl = process . argv [ 2 ] ; const sitemapUrl = process . argv [ 3 ] ; const parser = new XMLParser ( ) ; function normalize ( input , base ) { try { const url = new URL ( input , base ) ; url . hash = "" ; return url . href ; } catch { return null ; } } async function crawl ( start ) { const origin = new URL ( start ) .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.