A while ago I started getting this error in production: "payload too large" . The culprit was a Redis value that kept growing, a big JSON of vehicle data that we cache and read all the time.
My first thought was to bump the limit. My second thought was: why am I storing this much text in the first place? JSON is super repetitive. Same keys, over and over, for thousands of rows. That's basically a gift for compression.
So I tried gzip. It turned out to be one of the simplest fixes I've done, and it gave me more benefits than I expected.
π§© The Idea
Nothing fancy:
If the JSON is small, store it as usual.
If it's bigger than a threshold, gzip it, encode it as base64, and add a gz: prefix.
When reading, check the prefix. Has gz: ? Decompress. No prefix? Parse as usual.
That prefix is the trick that made this safe to ship. A valid JSON string never starts with gz: , so old entries already sitting in Redis still work.β¦