Book: LLM Observability Pocket Guide
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub
You've already had the conversation. Someone in the standup says "we need observability on the LLM side" and four browser tabs open. Langfuse. LangSmith. Phoenix. Arize. The first 30 minutes feel productive. By minute 45 the differences blur and you're back to comparing pricing pages.
In January 2026 ClickHouse acquired Langfuse , which nudged the market again. Self-host got more credible; the SaaS calculus shifted. Datadog shipped native OpenTelemetry GenAI semconv support in v1.37 around the same window. Your APM is in the conversation now.
This post is one experiment: same app, four backends. No anointed winner up front. The picks fall out of which trade-offs you can live with.
The app under test
Picture a boring RAG bot. FastAPI endpoint, pgvector retrieval over a 12k-document corpus, gpt-4o-mini for the answer.…