Menu

Your AI Agent Probably Belongs in the Data Pipeline
📰

Your AI Agent Probably Belongs in the Data Pipeline

DEV Community: database·elliot hutchins·about 5 hours ago
#uqHQEb4T
#devtodatabase#devto#dev#agent#agents
Reading 0:00
15s threshold

There's a question I keep coming back to with AI and data engineering:

Do we move the data to the agent, or move the agent to where the data already lives?

Because we're starting to put agents everywhere.

In the warehouse.

On the source databases.

Inside BI.

Writing transforms.

Watching queries.

Building DAGs.

Debugging pipelines.

And if you aren't careful, you end up with five agents, five sets of credentials, five different copies of context and a developer laptop somehow holding the whole thing together.

That doesn't feel like where this should end up.

Take a boring data stack

Nothing exotic.

You've got a handful of MSSQL databases.

Fivetran moves the data into Snowflake.

You've got some DAGs taking it through bronze, silver and gold.

Then BI sits on top.

Pretty normal.

Swap MSSQL for MongoDB, Snowflake for Databricks and Fivetran for Hevo if you want.

The names change.

The problem really doesn't.

Now add agents.

Where do they go?

The annoying answer is "everywhere"

Ask a data…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.