Menu

1 / 3

Your judge is drifting too — why evaluation standards move even when the model doesn't

DEV Community: machinelearning·Felipe 0liveira·4 days ago
#sbwxJ33J
Reading 0:00
15s threshold

This digest covers LLM evaluation and observability developments from 2026-11-05 through 2026-11-12. The headline thread this week: several independent papers converge on the same uncomfortable finding — the instruments we use to measure LLM quality (human raters, LLM-judges, contamination tests, failure taxonomies) are themselves unstable, and that instability is often bigger than the signal teams are trying to measure.

🔥 Highlights

AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models — human judgment drifts even when the model doesn't.

MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification — deterministic checks beat a VLM-judge on accuracy.

Langfuse launches AI Gateway — one proxy swap turns every model call into a trace.

Understand how agents use your MCP server — traces the silent bugs where a tool call "succeeds" with wrong data.

Gemini Decisions API — a third judge-as-a-service API lands, and its outputs…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.