Menu

My Snapchat scraper returned 0 results for a day and still said "SUCCEEDED"
📰

My Snapchat scraper returned 0 results for a day and still said "SUCCEEDED"

DEV Community: nextjs·Zait Dioses·about 1 hour ago
#mEO1nGWx
#devtonextjs#devto#dev#json#exit
Reading 0:00
15s threshold

On Monday I checked my Snapchat Spotlight scraper and it looked fine. Green status, run finished in five seconds, exit code 0.

The dataset was empty.

It had been empty since Snapchat pushed a change to their website, and nothing told me. No error, no failed run. If I hadn't opened the output by hand I would have kept "selling" a tool that returned nothing.

Here's what broke, how I found the data again, and the second bug I only noticed because of the first one.

What changed

Public Snapchat profiles ( snapchat.com/@username ) used to be a classic Next.js page. All the data sat in one script tag:

<script id = "__NEXT_DATA__" type = "application/json" > { . . . props . pageProps . . . } </script>

Enter fullscreen mode

Exit fullscreen mode

My parser grabbed that JSON and read spotlightHighlights and spotlightStoryMetadata . Simple.

This week the profile pages moved to the Next.js App Router. __NEXT_DATA__ is gone. My code raised an error, the loop caught it, logged it, and moved on to the next URL.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.