Menu

1 / 2

# How I Built a PDF Processing Pipeline Designed to Scale Toward 10k Users — FastAPI, Inngest & Smart Async Design

DEV Community·Mohamed Elsayed Ali·6 months ago
#c3xvrM6H
Reading 0:00
15s threshold

I recently built a PDF processing pipeline inside my project Playground .

The goal was simple:

Users upload PDFs, the API responds immediately, and heavy processing happens in the background without blocking request latency.

That sounds straightforward until traffic increases.

A naive implementation works fine for a few users — but starts falling apart when many uploads arrive at once.

In this article, I’ll walk through the architecture, the async design decisions, and a few lessons I learned while building it.

Repository: Playground on GitHub

The Problem

A common first implementation looks something like this:

receive upload

parse PDF inside the request

extract text

store results

return response

It works — until multiple users upload files concurrently.

The biggest issue is that PDF parsing is not purely async work .

Libraries like pdfplumber perform CPU-heavy parsing.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.