On Monday I checked my Snapchat Spotlight scraper and it looked fine. Green status, run finished in five seconds, exit code 0.
The dataset was empty.
It had been empty since Snapchat pushed a change to their website, and nothing told me. No error, no failed run. If I hadn't opened the output by hand I would have kept "selling" a tool that returned nothing.
Here's what broke, how I found the data again, and the second bug I only noticed because of the first one.
What changed
Public Snapchat profiles ( snapchat.com/@username ) used to be a classic Next.js page. All the data sat in one script tag:
<script id = "__NEXT_DATA__" type = "application/json" > { . . . props . pageProps . . . } </script>
Enter fullscreen mode
Exit fullscreen mode
My parser grabbed that JSON and read spotlightHighlights and spotlightStoryMetadata . Simple.
This week the profile pages moved to the Next.js App Router. __NEXT_DATA__ is gone. My code raised an error, the loop caught it, logged it, and moved on to the next URL.…