Menu

Post image 1
Post image 2
1 / 2
0

oh-my-agent: skills now measure and optimize their own utility

DEV Community: productivity·gracefullight·3 months ago
#vj4mkULz
#dev#slide#skill#skills#eval#default
Reading 0:00
15s threshold

Most skill libraries grow by accretion. You add a SKILL.md, it sounds useful, and it lives forever because nobody can prove it helps or hurts. This week oh-my-agent closed that gap: oma skills eval measures whether loading a skill actually improves held-out task outcomes, and oma skills opt rewrites the skill to push that number up. 194 commits landed, CLI is at 8.41.0, but the eval-to-opt loop is the part worth your attention. What's new oma skills eval : measures utilityLift (treatment vs baseline) on held-out tasks. --mock replays recorded rollouts deterministically, --live spawns two read-only agentic arms per task, --record captures the rollouts. Default checker is judge (an LLM grades output against a rubric); assert and regex are opt-in deterministic checks.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More