Menu

Before you migrate a website, save a list of every old URL (free Wayback Machine script)
📰

Before you migrate a website, save a list of every old URL (free Wayback Machine script)

DEV Community: python·Andy Zeng·18 days ago
#pDirsS7F
#devtopython#devto#dev#urllib#path
Reading 0:00
15s threshold

Disclosure: I build AppGripes and a small free migration page mentioned at the end. Everything else here works with free tools. This article was drafted with AI assistance; the code was tested on Oct 30, 2026.

The most expensive mistake in a site migration is quiet: old URLs that used to get traffic and backlinks start returning 404 on the new site, and nobody notices for weeks. Search Console only shows you some of them, and only after Google has crawled the broken pages.

The fix is boring and cheap: before you switch, write down every URL the old site ever had, then check each one against the new site and add a 301 for anything that breaks.

The catch is getting that list. Your CMS knows today's pages, not the /blog/2019/old-slug from three redesigns ago that a partner still links to. The Internet Archive does know, and it has a free API for it.

Step 1: list every archived URL with the Wayback CDX API

The CDX server returns one row per capture.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.