Reasoning eval
Is what the model says about the board actually true?
Models solve Lichess puzzles and write out their reasoning. Every factual claim in that reasoning is extracted and checked against python-chess and Stockfish. The number this eval exists to produce is the gap between how often a model picks the right move and how often the things it says about the position hold up.
A model can solve a puzzle while describing a pin that does not exist. Solve rate cannot see that — it reports one bit per puzzle. Checking the reasoning turns a single puzzle into a dozen verifiable assertions.
Puzzles
200
3 models, 600 traces
Engine
d20
Stockfish, one line per position
Spend
$24.38
Extractor: claude-haiku-4.5
Run
2026-09-08
commit 040532b
Headline
Gap = solve rate − claim accuracy, in points
| Model | Solve rate | Claim accuracy | Gap | Unverifiable | Claims/task | $/task |
|---|---|---|---|---|---|---|
| GPT-5.2 | 57.0%[50.0–64.0] | 57.8%[54.2–61.5] | −0.8 | 20.9% | 6.6 | $0.0575 |
| Claude Sonnet 5 | 26.5%[20.5–32.5] | 51.8%[47.4–55.9] | −25.3 | 26.5% | 5.0 | $0.0370 |
| Qwen3.8 A95B170/200 | 33.0%[27.0–39.5] | 40.0%[36.1–44.0] | −7.0 | 17.0% | 7.2 | $0.0274 |
Claim accuracy is true / (true + false); unverifiable claims are excluded from it and reported separately. Read it carefully — the metric rewards vagueness, because a model that hedges everything into unverifiable prose scores well on a very small denominator. Read it alongside claims-per-task and the unverifiable rate, never alone. A badge marks a model the verifier did not finish; its rates rest on the smaller denominator shown rather than on the full run.
What actually separates the models
Threshold: 5 points and no overlap
Claim accuracy spans 17.8 points, from 40.0% (Qwen3.8 A95B) to 57.8% (GPT-5.2). It does not separate them — intervals still overlap for GPT-5.2 / Claude Sonnet 5. The point estimates order these models; the data does not.
State-claim accuracy spans 19.1 points, from 67.3% (GPT-5.2) to 86.4% (Qwen3.8 A95B). It does not separate them — intervals still overlap for Qwen3.8 A95B / Claude Sonnet 5, Claude Sonnet 5 / GPT-5.2. The point estimates order these models; the data does not.
Line-claim accuracy spans 27.8 points, from 28.1% (Qwen3.8 A95B) to 55.9% (GPT-5.2). It separates the models — the spread clears 5 points and no intervals overlap.
A state claim is a fact about the position as it stands — “the knight on f6 is pinned.” A line claim is a calculated variation — “if Qxc4, Nxc4 wins the queen for a rook.” They are different abilities, and a single accuracy figure hides which one a model has. Intervals are 95% percentile bootstraps over 2000 resamples of whole tasks, not individual claims: claims from one trace are correlated, and resampling them independently would report an interval far narrower than the evidence supports.
How much evidence is behind each number
| Model | True | False | Unverifiable | State claims | Line claims | Plans naming a move | Sound plans |
|---|---|---|---|---|---|---|---|
| GPT-5.2 | 601 | 438 | 275 | 67.3%n=251 | 55.9%n=655 | 14% | 43% |
| Claude Sonnet 5 | 384 | 358 | 268 | 74.8%n=262 | 40.4%n=438 | 35% | 31% |
| Qwen3.8 A95B | 477 | 715 | 245 | 86.4%n=257 | 28.1%n=823 | 81% | 32% |
Read the last two columns together. Where a plan does not name its move, a model call has to infer one, and the resolver’s mistakes then read as the model contradicting itself — so a model at 14% is having most of its plans guessed at, and its figures are not comparable with one at 81%.
By rating band
Cut from cached data — no model calls, no engine runs
| Band | Claude Sonnet 5 | GPT-5.2 | Qwen3.8 A95B |
|---|---|---|---|
| 1200 | 55.8%n=181 | 68.2%n=277 | 39.7%n=343 |
| 1600 | 53.2%n=190 | 55.3%n=253 | 50.0%n=274 |
| 2000 | 48.0%n=179 | 53.4%n=264 | 34.3%n=315 |
| 2400 | 50.0%n=192 | 53.5%n=245 | 36.9%n=260 |
By motif
Motifs appearing in at least 10 tasks
| Motif | Claude Sonnet 5 | GPT-5.2 | Qwen3.8 A95B |
|---|---|---|---|
| advancedPawn | 53.5%n=71 | 55.8%n=120 | 48.2%n=114 |
| advantage | 52.6%n=272 | 53.3%n=353 | 42.7%n=419 |
| attraction | 36.8%n=68 | 56.1%n=66 | 46.5%n=99 |
| crushing | 51.6%n=335 | 55.4%n=491 | 37.5%n=550 |
| defensiveMove | 47.4%n=57 | 51.5%n=97 | 40.0%n=80 |
| deflection | 30.8%n=52 | 52.1%n=73 | 35.7%n=84 |
| endgame | 50.1%n=383 | 57.6%n=569 | 39.6%n=628 |
| exposedKing | 53.2%n=47 | 58.3%n=60 | 37.8%n=74 |
| fork | 50.0%n=106 | 49.2%n=124 | 33.1%n=154 |
| kingsideAttack | 47.4%n=57 | 66.7%n=66 | 44.9%n=89 |
| long | 52.2%n=318 | 54.1%n=436 | 41.5%n=544 |
| master | 52.9%n=102 | 52.2%n=138 | 29.1%n=117 |
| mate | 51.1%n=133 | 73.7%n=190 | 41.6%n=214 |
| mateIn2 | 48.6%n=74 | 79.6%n=103 | 35.2%n=122 |
| middlegame | 54.3%n=326 | 57.0%n=421 | 39.1%n=501 |
| pawnEndgame | 40.0%n=35 | 58.0%n=69 | 42.6%n=61 |
| pin | 53.5%n=43 | 56.5%n=62 | 35.5%n=62 |
| sacrifice | 43.4%n=99 | 48.7%n=113 | 51.7%n=118 |
| short | 53.0%n=315 | 61.2%n=459 | 36.5%n=482 |
| veryLong | 45.1%n=91 | 52.7%n=110 | 43.7%n=126 |
What this does not measure
Contamination is not controlled
The Lichess export carries no puzzle creation date — its only date column marks the day a puzzle was featured, and is absent for almost all of them. These puzzles may well appear in training data. Nothing here separates recall from reasoning.
The extractor is a model, not an oracle
Claims are extracted by claude-haiku-4.5, and a claim it fails to extract is one the model is never held to. Its own error rate is unmeasured and is included in none of the intervals above.
Plan scoring depends on resolving a sentence to a move
Where a plan names its move, that move is read from the text. Where it does not, a model call resolves it, and plans that resolve to nothing are excluded rather than assumed.
See it happen
Read the 600 traces these numbers came from
Every puzzle, every model's reasoning, and each claim marked true, false or unverifiable against the board it was made about. The table above is the summary; this is the evidence.
The other eval grades the moves rather than the words — see the arena scorecard.