I'm building Arcyria , a multi-tenant data operations platform: data connections, ELT pipelines, SQL transformations, monitoring and an AI operator, in one place. This post covers the architecture decision that shaped the most: every pipeline run is its own short-lived container.
The problem with long-running workers
The obvious way to run pipelines is a pool of workers that pull jobs from a queue. It works, until it doesn't:
One tenant's heavy run starves everyone else.
State leaks between runs (temp files, caches, half-loaded tables).
Credentials end up living in long-lived processes.
Scaling means guessing how many workers you need.
The approach: one run, one container
When a user starts a run, the API does not execute anything. It publishes a small, credential-free event to a queue.…