Long-running work should not sit inside an HTTP request. Sending a transactional email, generating a PDF, processing an upload, or embedding a batch of documents for a search or RAG pipeline can take seconds or minutes. If that work runs in the request-response cycle, users wait, workers time out, and a retry from the client can start the same job twice.
Python gives you several ways to move that work into a background job. The useful question is not “which tool is best,” but which one matches the shape of your app: a small FastAPI service, a growing backend with Redis, or a distributed system that needs retries, schedules, and multiple queues.
This guide compares four options you will actually see in Python backends: FastAPI BackgroundTasks , RQ, ARQ, and Celery. It stays on the job-queue choice itself, not general async syntax or concurrency models.
What a background job needs to get right
A queue is more than “run this later.” For anything you care about, the setup should give you:
Durability.…