I recently built a PDF processing pipeline inside my project Playground .
The goal was simple:
Users upload PDFs, the API responds immediately, and heavy processing happens in the background without blocking request latency.
That sounds straightforward until traffic increases.
A naive implementation works fine for a few users — but starts falling apart when many uploads arrive at once.
In this article, I’ll walk through the architecture, the async design decisions, and a few lessons I learned while building it.
Repository: Playground on GitHub
The Problem
A common first implementation looks something like this:
receive upload
parse PDF inside the request
extract text
store results
return response
It works — until multiple users upload files concurrently.
The biggest issue is that PDF parsing is not purely async work .
Libraries like pdfplumber perform CPU-heavy parsing.…