If you track a competitor's pages, a vendor's changelog, or the stack behind a domain, you probably have a cron job somewhere that fetches something, diffs it against last time, and pings you. It works until it doesn't: you store snapshots, handle sitemap index files, normalize three feed formats, and decide what "changed" means.

This post walks through a push-based version of that loop: tell a service what to watch, and receive a signed webhook when something changes. I'll use Crawlwise Data , which I work on (disclosure at the end). The pattern is the same whichever tool you use.

Three kinds of change worth watching

Sitemap changes. Diffing a site's XML sitemap shows new, updated ( lastmod ), and removed URLs. One catch: if you only fetched part of a large sitemap, a URL missing from your partial result is not a removal. A diff that treats truncation as deletion will send you false "removed" alerts, so be careful here.

Feed items. RSS, Atom, RDF, and JSON Feed all describe "new post" differently.…