Menu

How to score LLM forecasts honestly: our Claude Opus swarm vs Polymarket, resolved
📰

How to score LLM forecasts honestly: our Claude Opus swarm vs Polymarket, resolved

DEV Community: datascience·ekocam·5 days ago
#T2akV4wW
#devtodatascience#science#devto#dev#market
Reading 0:00
15s threshold

Two weeks ago we posted forecasts from a MiroFish agent swarm running on Claude Opus 5.5, from Opus, Sonnet and Haiku asked directly, and from wallet positioning data, on four live Polymarket markets. Every number was pushed to GitHub before any market resolved. All four have now resolved, and the full results are on our blog with the scoreboard on GitHub .

Short version: the market price beat every AI source. This post is about the scoring itself, because most "my model predicts X" demos skip it.

1. Lock the forecasts before the outcome

The whole exercise is worthless if the numbers can move after the fact. We committed forecasts.csv to a public repo on October 25; GitHub's event log records the push time, and the repo went public before the first market resolved. If you run your own test, a timestamped commit is the cheapest proof you can get.

2.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.