At EMNLP 2026 in Budapest, I presented a poster with my colleagues. The paper behind it asks a simple question. When a vision-language model fails at a visual puzzle, did it fail to plan, or did it fail to see the board? When I talked to people at the poster, the conversations kept turning to a more practical version of that question. When a model does badly on an image task, what should you change first?
The usual first move is the prompt. You reword the instructions, add an example, or change the format. Our results suggest that for image inputs, this is often the wrong place to start. The image matters more. But piling more text onto the image does not help either. What helped was labeling the few things the model could not see, in the same terms the task uses.
The setup
SPaRC is our benchmark of 1,000 grid puzzles, inspired by the video game The Witness . The model has to draw one path from a start to an exit that follows every rule on the board. It writes the answer as a list of (x, y) coordinates.…