People in our study solved 98.0% of the puzzles in this post. An easy one took them about 11 seconds and the hardest ones a little over two minutes. The best model we have tested, GPT-6.1 Sol, solves 83.6% of them. That sounds close to human. On the hardest level, though, it solves 49.4%, while people solve 94.5%.

Solving one of these puzzles takes several separate skills, and a single accuracy number hides which of them a model lacks. So we took the task apart and asked four questions.

Can models read the board, especially from an image?

Can they keep track of their own line?

Can they follow the rules?

Do they try harder when a puzzle is harder?

This post brings together key results from three of our papers, listed below. BibTeX entries for all three are on the project website .

Lars Benedikt Kaesberg, Jan Philip Wahle, Terry Ruas, Bela Gipp. SPaRC: A Spatial Pathfinding Reasoning Challenge . EMNLP 2025.

Lars Benedikt Kaesberg, Tianyu Yang, Niklas Bauer, Terry Ruas, Jan Philip Wahle, Bela Gipp.…