BudgetMix RL: What Multi-Turn Reinforcement Learning Really Improves in LLM Agents

Teams training LLM agents with multi-turn reinforcement learning usually report one number: success rate on a long-horizon benchmark. A new paper, What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents? , asks what sits behind that number. Its answer matters for anyone shipping agents: the gains depend heavily on how many turns the agent is allowed at inference time, and they come from a specific skill, recovering from mistakes. The authors also propose BudgetMix RL, a training change meant to make agents robust when the turn budget shrinks.

Why a single success rate hides the story

Agentic RL trains a model on whole interaction trajectories: the model reasons, calls a tool, reads the result, and repeats until it finishes or runs out of turns.…