Menu

Building Arcyria: an ELT platform where pipelines run as one-shot containers
📰

Building Arcyria: an ELT platform where pipelines run as one-shot containers

DEV Community: go·Vijay Dabhi·5 days ago
#SVgOYjkA
#devtogo#devto#dev#task#tables
Reading 0:00
15s threshold

I'm building Arcyria , a multi-tenant data operations platform: data connections, ELT pipelines, SQL transformations, monitoring and an AI operator, in one place. This post covers the architecture decision that shaped the most: every pipeline run is its own short-lived container.

The problem with long-running workers

The obvious way to run pipelines is a pool of workers that pull jobs from a queue. It works, until it doesn't:

One tenant's heavy run starves everyone else.

State leaks between runs (temp files, caches, half-loaded tables).

Credentials end up living in long-lived processes.

Scaling means guessing how many workers you need.

The approach: one run, one container

When a user starts a run, the API does not execute anything. It publishes a small, credential-free event to a queue.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.