Menu

We Stopped Installing LLM SDKs in Every Microservice
πŸ“°

We Stopped Installing LLM SDKs in Every Microservice

DEV Community: microservicesΒ·AqibΒ·10 days ago
#XK1S8nPF
#devtomicroservices#devto#dev#vendor#model
Reading 0:00
15s threshold

We almost put the OpenAI SDK inside the agent runtime.

It looked like the short path. The runtime already runs a turn: load the published agent, load the chat, maybe search knowledge, maybe load memory, call a model, save the assistant message. Installing @anthropic-ai/sdk or the OpenAI client right there would have made the first demo work in an afternoon.

We did not do that. This post is about why, what we built instead, and how that choice helps a multi-tenant agent platform when the traffic is no longer a laptop.

The audience I have in mind is a senior backend or platform engineer who has already shipped one LLM feature and is now asking where the keys, the retries, and the bill should live.

What we were building

QuickStart AI is a production-oriented platform. Organizations run workspaces . A workspace is the tenant. Every customer row is scoped by workspace_id .

Inside a workspace we already had the product pieces a chat agent needs:

Agents. Instructions and model settings. Publish is immutable.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.