Arena · Scorecard
How well do language models actually play chess?
Every model answers the same positions, alone — there is no opponent whose choices could skew anyone’s score. Stockfish grades each move against its own best, and what comes out is the board below.
Evidence so far
Gemini 3.7 Flash scores highest at 81.5%, but its confidence interval overlaps GPT-5 nano’s. At 30 positions, this sample orders the two — it does not separate them. Read the gap between them as noise until the set grows. Across all 3 models, the share of moves that throw away a winning or level position runs from 16.7% to 43.3%.
30 positions from core-v1, answered under the v1-neutral prompt with legal moves shown, graded by Stockfish 18 Lite WASM at depth 14. Set hash 0e1484c4.
| Model | n | ||||||
|---|---|---|---|---|---|---|---|
Gemini 3.7 Flash | 81.5% ± 9.7% | 92 | 46.7% | 16.7% | 4.26p | $0.0023 | 30 |
GPT-5 nano | 71.0% ± 12.0% | 181 | 40% | 26.7% | 3.28p | $0.0009 | 30 |
Claude Haiku 4.5 | 57.1% ± 12.9% | 269 | 26.7% | 43.3% | 5.52p | $0.0064 | 30 |
Rows are ranked by the sorted column. Click one for its per-category breakdown.
Where the positions come from
Engine self-play, not books or puzzle databases. Every position is novel by construction, so no model can have seen it in training — contamination is designed out rather than defended against. Sets are balanced on category and side to move.
Why the error bars matter
Each score carries a 95% confidence interval from a seeded bootstrap. Where two intervals overlap, the data does not separate those models, and the board says so rather than implying a ranking it cannot support.
Reproducibility
Every row is traceable to a position-set hash, a frozen prompt hash, and a pinned engine build and depth. Scores from different conditions are never mixed — they are shown as separate boards.
The other eval
A good move is not the same as a true explanation
This board scores the move. The reasoning eval scores the sentences a model writes about the position — every claim checked against python-chess and Stockfish.