# The numbers behind each finding

Source: https://plannerbench.org/results.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt

Results

Each section below backs up one line of the [home page](https://plannerbench.org/), with the short table and a link to the full detail. How the questions were built and checked, and how the tests differ, is on [Technical details](https://plannerbench.org/method.html).

Finding 1 · First test · October 2026

## One rule at a time, it’s great

A **one-rule question** applies a single rule, like the tax on an income or this year’s required IRA withdrawal.

| Model | One rule | Several rules at once |
|---|---|---|
| Claude Sonnet 5.5 (Anthropic) | 97% (89–99) | 37% (22–54) |
| Claude Haiku 5.5 (Anthropic) | 97% (89–99) | 17% (7–34) |
| DeepSeek V4.1 Flash (DeepSeek · free to download) | 86% (77–91) | 36% (23–50) |
| GLM 5.3 Flash (Z.ai · free to download) | 68% (58–77) | 27% (16–41) |
| gpt-oss-120b (OpenAI · free to download) | 36% (26–46) | 5% (2–18) |

Claude Sonnet 5.5 got 97% of one-rule questions right, and Claude Haiku 5.5 got 97%. When several rules met in one household, Sonnet dropped to 37%. The shaded band behind each bar is the range the true score likely sits in.

[The first test in full →](https://plannerbench.org/alone.html#october)

Finding 2 · Newer, harder set · November 2026

## Several rules at once, it depends heavily on which AI

42 harder questions, each making two or more rules meet: the largest Roth conversion that won’t raise Medicare premiums two years later, a widow’s first year of taxes, Arizona state tax for retirees. 11 models, at low reasoning effort with a 4,000-token reply limit.

| Model | Right, of the newer set |
|---|---|
| GPT-6.1 Sol (OpenAI · top-priced) | 98% · 41/42 (88–100) |
| GPT-6.1 Terra (OpenAI · mid-priced) | 80% · 33.5/42 (65–89) |
| Claude Sonnet 5.5 (Anthropic · top-priced) | 62% · 26/42 (47–75) |
| DeepSeek V4.1 Flash (DeepSeek · cheap and free) | 29% · 12/42 (17–44) |
| Llama 4 Maverick (Meta · cheap and free) | 6% · 2.5/42 (2–17) |

GPT-6.1 Sol got 98%, GPT-6.1 Terra 80%, Claude Sonnet 5.5 62%, and all 5 cheap and free models under 30%. These settings differ from the October test, so don’t compare the two sets of scores.

[All 11 models in full →](https://plannerbench.org/alone.html#november)

Finding 2 · The ChatGPT model · November 2026

## GPT-6 Luna

GPT-6 Luna, asked the same 42 hard questions through OpenAI’s Codex app on my ChatGPT subscription, with code and web off, graded by the same answer key.

| Questions | GPT-6 Luna |
|---|---|
| First test · one rule | 71% · 42 of 59 |
| Newer, harder set | 17% · 7 of 42 |
| Wrong facts · caught the planted mistake | 50% · 15 of 30 |

It ran with more time to think and no length limit, in a different setup from the models above, so it isn’t ranked with them. Why, and how the run was done, is on [Technical details](https://plannerbench.org/method.html#chatgpt).

[The GPT-6 Luna run in full →](https://plannerbench.org/alone.html#chatgpt)

Finding 2 · Calculator test · October 2026

## A calculator that only gave the number didn’t fix it. One that showed the rule did.

Claude Haiku 5.5 on the 50 hardest first-test questions, asked three ways: on its own; with a calculator that sends back only the number; and with one that also shows the rule and the IRS form line behind it.

**Result.** 40% on its own, 62% with only the number, 100% with the rule shown. On 50 held-back questions nobody tuned against: 44% → 96%.

| Claude Haiku 5.5 | The hardest questions | Held-back questions | Dollars off per question, per year |
|---|---|---|---|
| On its own | 40% · 20/50 (28–54) | 44% · 22/50 (31–58) | $318 (held-back $216) |
| Calculator 1.0: just the number | 62% · 31/50 (48–74) | (not asked) | $229 |
| Calculator 1.1: the number, the rule and its source | 100% · 50/50 (93–100) | 96% · 48/50 (87–99) | $0 (held-back $4) |

The same 50 questions in every row (5 tasks, 10 each). The right number alone fixed 11 questions it had wrong and broke 0 it had right; adding the rule and its source fixed 19 more and broke 0. With the same questions asked both ways, a jump that size is very unlikely to be luck (the statistics are under “For the technical reader” below). On 50 held-back questions, never seen while the calculator was built: 26 fixed, 0 broken. The shaded band behind each bar is the range the true score likely falls in, given so few questions.

**Not every model was rescued.** On the newer set, DeepSeek V4.1 Flash rose from 29% to 79% with the calculator; three other cheap models stayed at 10% or less.

[All three calculator tests in full →](https://plannerbench.org/calculator.html)

Finding 3 · Wrong facts · November 2026

## It believes whatever you tell it

30 client files, each with one planted mistake (two digits swapped in a balance, a birth year that makes a retiree 14, monthly spending far beyond what the family owns). A model **caught** it only if it named the mistake *and* stopped to ask instead of giving a number.

| Of 30 planted mistakes | Recognized it | Stopped and asked | Caught (both) | Refused a fine file |
|---|---|---|---|---|
| GPT-6.1 Sol (top-priced) | 100% | 80% | **80%** | 3% |
| Claude Sonnet 5.5 (top-priced) | 93% | 73% | **73%** | 0% |
| Gemini 3.8 Flash (mid-priced) | 92% | 100% | **92%** | 50% |
| DeepSeek V4.1 Flash (cheap and free) | 76% | 68% | **61%** | 0% |
| gpt-oss-120b (cheap and free) | 3% | 7% | **3%** | 5% |

Even the best at catching mistakes without false alarms, GPT-6.1 Sol and Claude Sonnet 5.5, missed 20–27%. Three of the cheap and free models (Mistral Small, gpt-oss-120b and Llama 4 Maverick) caught 1–3%. An AI judge decided whether a reply named the mistake; how that was checked is on [Technical details](https://plannerbench.org/method.html#ai-work).

One example

A client file said a retired school administrator with a $785,000 IRA was born in 2012, and asked how much she must withdraw this year. One model (gpt-oss-120b) replied:

> “Priya will be 14 years old throughout 2026 … no required minimum distribution is required. ANSWER: 0”

It saw that she was 14 and kept going. With her real birth year, the answer is $34,279.

[The wrong-facts test in full →](https://plannerbench.org/wrong-facts.html)

[Next **How I built it**](https://plannerbench.org/built.html)

---

PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com)

Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown.

Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice.
