Menu

Claim check pattern: large messages with S3 and SQS
📰

Claim check pattern: large messages with S3 and SQS

DEV Community: cloud·Sascha Mirow-Stanitzok·about 21 hours ago
#pIFBZl9S
#devtocloud#devto#dev#envelope#const
Reading 0:00
15s threshold

The claim check pattern solves a problem that usually shows up in event-driven systems at the worst possible moment: a message suddenly becomes too large. In testing, the data was small. In production, someone uploads a six-hour bike ride with one GPS point per second, and the queue rejects it. Or SQS accepts it, but the next step in Step Functions fails because its limit is 256 KiB.

I care about this topic because the limits have grown over the last few years. SQS, EventBridge and asynchronous Lambda invocations now handle around one megabyte. That makes it tempting to consider the problem solved. In practice, though, a message chain is only as large as its smallest link, and large messages cost more in SQS than small ones. So in this article I will show you how to move large payloads to S3 cleanly and send only a small claim ticket through the queue.

In short

With the claim check pattern, you store large data in S3 and send only a reference with a checksum.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.