Many teams store an AI agent's past attempts, each marked with a score, and show them to the model on the next try. The score is a reward label. The hope is that the model reads it and repeats what worked. I tested that hope by flipping the labels.

In my test, each trial gave a model a made up shipping order and asked it to choose one of four delivery routes, named Kiwi, Plum, Fig, and Lime. A hidden rule decided which route was right, and the model had to work that rule out from its memory.

I compared the first six trials of each run with the last six. By the last six, GPT-6 Sol answered 94.4 out of 100 correctly with true labels and 7.8 with flipped ones. Claude Haiku 5.5 rose from 21.1 to 37.2 with true labels. Qwen3 4B Instruct stayed near 32.8.

A paper submitted on 8 October 2026, Do LLMs Learn from Rewards in Context? , reports something different. Its abstract says that flipping, randomizing, or removing rewards changed improvement curves very little across four benchmarks and six models.…