Two ways to measure a model over a chessboard
Most model comparisons are vibes. Chess has a referee that is stronger than every player in the room, which turns an opinion into a scoreboard — once for the moves a model chooses, and again for the things it claims are true about the position in front of it.
Arena
Two models play. Stockfish grades every move.
Watch a full game unfold with both models' analysis streaming beside the board, an eval bar that moves as they blunder, and a scorecard that keeps count. Then read the frozen-position benchmark, where every model answers the same positions alone.
81.5%
Best move accuracy, Gemini 3.7 Flash, over 30 frozen positions.
Reasoning eval
Models explain themselves. Every claim gets checked.
Models solve Lichess puzzles and write out their reasoning. Each factual assertion in it is extracted and verified against python-chess and Stockfish, so a confident sentence about a defender that is not there is caught and counted.
40–58%
Share of what a model says about the board that is actually true, across 3 models and 200 puzzles.
One bit per game is not a measurement
Who won tells you almost nothing — a single game is a coin flip. Both evals here refuse that trade: the arena grades all ~40 moves a model makes, and the reasoning eval pulls roughly six checkable claims out of every puzzle.
The referee is stronger than the players
Stockfish is far beyond any language model at chess, so its verdict is ground truth rather than another opinion. That is the property that makes a scoreboard possible at all, and it is the thing most model comparisons lack.
Picking well and explaining well come apart
A model can find the right move while describing a pin that is not on the board, or reason soundly and then play something else. Two evals, because those are two abilities and one number cannot hold both.