Finding 2 · With a calculator
Give it a calculator it can trust, and some models get it right. Not all.
The plain answer
I turned my answer key into a calculator an AI model can use while it answers. It helps a lot, but only if the calculator shows its work and the model actually uses it well. It rescued some models, including a cheap one, and did nothing for others.
One example: it argued with the calculator
Claude Haiku 5.5 asked the calculator for a couple’s tax, got the right number, and replied:
“The calculation tool's figure of $21,639 applies the senior-deduction phase-out to each spouse separately, which I don't think the statute supports.”
It used its own wrong reading of a new 2025 tax break for people over 65 instead, and got the tax wrong. When the calculator also showed the rule and the IRS form behind the number, it stopped arguing.
What I mean by a “calculator”
In AI, a “tool” is a program a model can call in the middle of an answer, the way you’d reach for a calculator. Mine is the planning engine: code for 2026 federal tax, Social Security, required withdrawals from retirement accounts and Medicare premiums, checked against IRS, Social Security and Medicare publications. The model still has to decide what to type in and how to put the pieces together, which is where it can still go wrong.
Three calculator tests on this page
- Calculator test 1: number vs rule (October). One model, the hardest first-test questions. A right number wasn’t enough; the rule behind it was.
- Calculator test 2: who it rescues (November). Eight models on the newer, harder set. A cheap model jumped; three others didn’t move.
- Calculator test 3: whole-household decisions. Real choices over 25 years, three models. Plus GPT-6 Luna through Codex (the model on my ChatGPT subscription) without a calculator, and a short test: leave a fact out, and see if it asks.
Calculator test 1 of 3 · October 2026
Hand it the right number, and it argues. Show it the rule, and it listens.
- Questions
- the 50 hardest from the first test, plus 50 held-back ones
- Model
- Claude Haiku 5.5
- Asked
- three ways: on its own; with calculator 1.0 (sends back only the number); with 1.1 (the number, plus the rule and the IRS form line behind it)
- Settings
- high reasoning effort (lets the model think longer), long replies, one run per question
- Graded
- by code, against the answer key, within $1–2
- Held out
- yes: the held-back questions were kept secret and never used while building the calculator
Result. 40% on its own, 62% with only the number, 100% with the rule shown. On the held-back questions: 44% → 96%. The calculator’s numbers were the same in both versions; only what it said about them changed.
| Claude Haiku 5.5 | The hardest questions | Held-back questions | Dollars off per question, per year |
|---|---|---|---|
| On its own | 40% · 20/5028–54 | 44% · 22/5031–58 | $318held-back $216 |
| Calculator 1.0: just the number | 62% · 31/5048–74 | not asked | $229 |
| Calculator 1.1: the number, the rule and its source | 100% · 50/5093–100 | 96% · 48/5087–99 | $0held-back $4 |
The same 50 questions in every row (5 tasks, 10 each). The right number alone fixed 11 questions it had wrong and broke 0 it had right; adding the rule and its source fixed 19 more and broke 0. With the same questions asked both ways, a jump that size is very unlikely to be luck (the statistics are under “For the technical reader” below). On 50 held-back questions, never seen while the calculator was built: 26 fixed, 0 broken. The shaded band behind each bar is the range the true score likely falls in, given so few questions.
The question from the example at the top
- The calculator’s answer (and the key)
- $21,639
- Haiku with calculator 1.0
- $21,121Wrong
- Haiku with calculator 1.1
- $21,639Right
What version 1.1 sent back with the tax
- Income (MAGI) over the $150,000 threshold$39,225.00
- Reduction per person (6%)$2,353.50
- Senior deduction per person$3,646.50
- People 65 or older2
- Senior deduction$7,293.00
The senior deduction from the 2025 tax law (OBBBA), IRS Form 1040 Schedule 1-A Part V (lines 35-36b): $6,000 for each person 65 or older, reduced by 6% of MAGI over $150,000. The reduction is figured for each qualifying person separately: on a joint return each spouse's $6,000 is reduced by the full 6% of the excess, not one reduction off a combined $12,000.
- A right number wasn’t enough. Across all 140 questions, calculator 1.0 lifted Haiku from 77% to 84% (13 fixed, 3 broken, p = 0.021); single-step questions were already at 97%. But 18 of the 22 misses left were Haiku getting the right number from the calculator and replacing it with its own, every time saying so.
- It held on questions nobody tuned against. On 50 questions held back from a secret seed and never used while building the calculator: 44% on its own, 96% with calculator 1.1. Both misses were Haiku overruling the calculator again, with the rule in front of it.
- Limits. One model, one run per question. The held-back questions came from the same generator, so they test the same kinds of rules. These are the hardest questions, which is why Haiku scores lower here than it does overall.
Calculator test 2 of 3 · November 2026
Who it rescues: a cheap model jumped, three others didn’t move
- Questions
- the 42 questions in the newer, harder set, each with several rules meeting in one household
- Models
- 8 of the 11 models, on their own and with the calculator (Claude Sonnet 5.5 and the two Gemini models weren’t run with it, to save money)
- Settings
- low reasoning effort; replies capped at 4,000 tokens (a token is about three-quarters of a word); with the calculator, the cap applied to each of up to 8 back-and-forth turns
- Compared
- one run with the calculator against the first run on its own, question by question
- Graded
- by code, against the answer key, within $2–5
- Held out
- no: the newer set has no secret questions
Result. DeepSeek V4.1 Flash, a cheap model, went from 29% to 79% with the calculator setup: 21 questions fixed, none broken. That’s higher than Claude Sonnet 5.5 on its own (62%), at about 1/15 of the cost per right answer ($1.18 per 1,000 right answers against $17.33).
Not all of that gain is the calculator. The setup also let a model write more in total, because the reply cap applied to each turn. On its own, DeepSeek hit the cap before answering on 71% of questions; with the calculator, 9 of its replies went past 4,000 tokens in total. So part of the jump may come from that extra room. The conservative number: counting only replies that stayed within 4,000 tokens in total, it got 27/42 (64%). Higher than Sonnet’s score, but on only 42 questions that lead could still be luck.
| Model | On its own (first run) | With the calculator (one run) |
|---|---|---|
| GPT-6.1 Soltop-priced · one run each way | 98% · 41/4288–100 | 95% · 40/4284–99 |
| DeepSeek V4.1 Flashcheap and free · one run each way | 29% · 12/4217–44 | 79% · 33/4264–88 |
| Claude Haiku 5.5mid-priced · one run each way | 48% · 20/4233–62 | 62% · 26/4247–75 |
| gpt-oss-120bcheap and free · one run each way | 7% · 3/422–19 | 10% · 4/424–22 |
| Claude Sonnet 5.5top-priced · reference: not run with the calculator | 62% · 26/4247–75 | not run |
Four of the 8 models run both ways, plus Claude Sonnet 5.5 on its own for reference; all of them are in the first table under “All models and columns” below. Both columns are one run per question, on the same 42 questions. Highlighted: the calculator added more than 30 points. The small grey numbers and the shaded band behind each bar show the range the true score likely falls in, given so few questions. These settings differ from calculator test 1 above, so don’t compare the two sets of scores.
Called it, then wrong
A single retiree’s tax, with a pension, Social Security, and a required withdrawal partly given to charity. gpt-oss-120b made one call with the pension alone, left out the withdrawal, and replied with one line:
“ANSWER: 1115”
The right answer is $9,614.
Right parts, wrong assembly
GPT-6.1 Sol took the federal numbers from the calculator for an Arizona tax question, then did the state part itself:
“Arizona’s standard deduction does not include the federal age-65 addition… $17,900 × 2.5% = $447.50. ANSWER: 448”
It missed Arizona’s separate $2,100 exemption for people 65 and older. The right answer is $395. The calculator gave it correct parts; putting them together was its own work.
- Three cheap models called it and stayed low. gpt-oss-120b, Llama 4 Maverick and Mistral Small called the calculator on 79% of the questions and still got only 2–4 of 42 right: wrong numbers in, or one call where the question needed three.
- The highest scorer gained nothing. GPT-6.1 Sol got 41/42 on its own and 40/42 with it.
- Claude Haiku 5.5 skipped it half the time. It called the calculator on 50% of questions, and once said so outright: “I did not run this through the calculator tools.” Its gain is small enough to be luck.
All models and columns (for the detail-minded)
Every model run both ways, one run each. Fixed: wrong on its own, right with the calculator; broken: the reverse. Called it: the share of questions where the model used the calculator at all.
| Model | On its own (first run) | With the calculator (one run) | Fixed / broken | Called it |
|---|---|---|---|---|
| GPT-6.1 Soltop-priced · one run each way | 98% · 41/4288–100 | 95% · 40/4284–99 | 0 / 1 | 79% |
| GPT-6.1 Terramid-priced · one run each way | 81% · 34/4267–90 | 86% · 36/4272–93 | 3 / 1 | 81% |
| DeepSeek V4.1 Flashcheap and free · one run each way | 29% · 12/4217–44 | 79% · 33/4264–88 | 21 / 0 | 90% |
| Claude Haiku 5.5mid-priced · one run each way | 48% · 20/4233–62 | 62% · 26/4247–75 | 8 / 2 | 50% |
| GLM 5.3 Flashcheap and free · one run each way | 17% · 7/428–31 | 43% · 18/4229–58 | 11 / 0 | 86% |
| gpt-oss-120bcheap and free · one run each way | 7% · 3/422–19 | 10% · 4/424–22 | 2 / 1 | 79% |
| Mistral Small (2603)cheap and free · one run each way | 7% · 3/422–19 | 7% · 3/422–19 | 3 / 3 | 79% |
| Llama 4 Maverickcheap and free · one run each way | 7% · 3/422–19 | 5% · 2/421–16 | 0 / 1 | 79% |
| Claude Sonnet 5.5top-priced · reference: not run with the calculator | 62% · 26/4247–75 | not run | – | – |
Every model in the November set, with every run averaged:
| Model | On its own | With the calculator | Called it | Cost per 1,000 right, on its own → with it |
|---|---|---|---|---|
| GPT-6.1 Soltop-priced · one run each way | 98%88–100 | 95%84–99 | 79% | $3.22 → $3.19 |
| GPT-6.1 Terramid-priced · on its own: 2 runs averaged · with it: 1 run | 80%65–89 | 86%72–93 | 81% | $1.89 → $1.79 |
| DeepSeek V4.1 Flashcheap and free · on its own: 2 runs averaged · with it: 1 run | 29%17–44 | 79%64–88 | 90% | $3.64 → $1.18 |
| Claude Sonnet 5.5top-priced · 1 run per item | 62%47–75 | not run | – | $17.33 |
| Claude Haiku 5.5mid-priced · on its own: 2 runs averaged · with it: 1 run | 52%38–67 | 62%47–75 | 50% | $2.21 → $3.63 |
| GLM 5.3 Flashcheap and free · on its own: 2 runs averaged · with it: 1 run | 15%8–29 | 43%29–58 | 86% | $6.30 → $2.46 |
| Gemini 3.1 Protop-priced · 1 run per item | 29%17–44 | not run | – | $51.35 |
| Gemini 3.8 Flashmid-priced · 1 run per item | 29%17–44 | not run | – | $19.47 |
| gpt-oss-120bcheap and free · on its own: 2 runs averaged · with it: 1 run | 7%2–19 | 10%4–22 | 79% | $1.60 → $5.02 |
| Mistral Small (2603)cheap and free · on its own: 2 runs averaged · with it: 1 run | 7%2–19 | 7%2–19 | 79% | $8.73 → $11.75 |
| Llama 4 Maverickcheap and free · on its own: 2 runs averaged · with it: 1 run | 6%2–17 | 5%1–16 | 79% | $18.47 → $62.65 |
Unlike the table above, “On its own” here averages every run of a question (two for most models, one for the most expensive), so Claude Haiku 5.5 shows 52% here and 20/42 (its first run) above. Each row says how many runs went into each column. Price group is by the model’s price per token: top-priced, mid-priced, or cheap and free.
By kind of question (for the detail-minded)
| Model | Roth conversion, Medicare-safe | Roth conversion cost with Arizona tax | Taxable Social Security | Inherited IRA withdrawal | Charity from a required withdrawal | Widow’s first year | Arizona tax |
|---|---|---|---|---|---|---|---|
| GPT-6.1 Soltop-priced · on its own → with it | 100% → 100% | 100% → 100% | 100% → 100% | 100% → 100% | 100% → 100% | 100% → 100% | 83% → 67% |
| GPT-6.1 Terramid-priced · on its own → with it | 100% → 100% | 75% → 100% | 100% → 100% | 92% → 83% | 92% → 100% | 100% → 100% | 0% → 17% |
| DeepSeek V4.1 Flashcheap and free · on its own → with it | 67% → 100% | 0% → 83% | 100% → 100% | 33% → 50% | 0% → 100% | 0% → 83% | 0% → 33% |
| Claude Sonnet 5.5top-priced · on its own | 100% | 17% | 100% | 83% | 67% | 67% | 0% |
| Claude Haiku 5.5mid-priced · on its own → with it | 100% → 100% | 8% → 50% | 92% → 83% | 33% → 50% | 67% → 83% | 67% → 67% | 0% → 0% |
| GLM 5.3 Flashcheap and free · on its own → with it | 42% → 100% | 0% → 0% | 33% → 50% | 25% → 33% | 0% → 67% | 8% → 50% | 0% → 0% |
| Gemini 3.1 Protop-priced · on its own | 17% | 0% | 100% | 83% | 0% | 0% | 0% |
| Gemini 3.8 Flashmid-priced · on its own | 0% | 0% | 100% | 67% | 33% | 0% | 0% |
| gpt-oss-120bcheap and free · on its own → with it | 0% → 0% | 0% → 17% | 17% → 0% | 33% → 33% | 0% → 17% | 0% → 0% | 0% → 0% |
| Mistral Small (2603)cheap and free · on its own → with it | 0% → 0% | 0% → 0% | 17% → 50% | 25% → 0% | 8% → 0% | 0% → 0% | 0% → 0% |
| Llama 4 Maverickcheap and free · on its own → with it | 0% → 0% | 0% → 0% | 8% → 0% | 33% → 33% | 0% → 0% | 0% → 0% | 0% → 0% |
Six questions of each kind. Each cell is right on its own (every run averaged) → right with the calculator. The kinds: the largest Roth conversion (moving money from a regular IRA to a Roth IRA and paying the tax now) that keeps Medicare premiums level; what a conversion costs in federal plus Arizona tax; how much Social Security is taxed when there’s tax-free bond interest and a gift to charity from an IRA; this year’s required withdrawal from an inherited IRA; tax when part of a required withdrawal goes to charity; a widow’s first tax year; Arizona tax for retirees.
Calculator test 3 of 3 · Whole-household decisions
Real retirement decisions: with the calculator, all three models made the right decision, with every number within 2%, every time
- Questions
- 11 whole made-up households (about 16 facts each: Social Security, IRAs, a pension, savings), one from each of 10 kinds of decision plus a trap with no honest answer
- Models
- Claude Sonnet 5.5, Claude Haiku 5.5, DeepSeek V4.1 Flash
- Settings
- high reasoning effort; reply budget 64,000 tokens for the Claude models and 32,000 for DeepSeek; one try each
- Graded
- by code: “fully right” means the right decision and every number within 2% of the key
- Held out
- no: 33 held-back questions are built but not yet run
Result. Each choice is one people really face: how much to convert to a Roth, when to claim Social Security, how to give to charity, how fast to empty an IRA inherited from someone who died. Getting it right means combining at least three rules over 25 years. With the calculator, all 3 models got every question fully right: the right decision, and every number within 2% of the key. On its own, DeepSeek V4.1 Flash got 2/11.
| Model | Fully right | Ran out of room | Thinking per question | Cost per question |
|---|---|---|---|---|
| Claude Haiku 5.5, on its own | 36% · 4/1115–65 | 0 | 34k tokens | $0.02 |
| Claude Sonnet 5.5, on its own | 70% · 7/1040–89 | 3 | 37k tokens | $0.38 |
| DeepSeek V4.1 Flash, on its own | 18% · 2/115–48 | 0 | 27k tokens | $0.01 |
| Claude Haiku 5.5 + calculator | 100% · 11/1174–100 | 0 | 9k tokens | $0.01 |
| Claude Sonnet 5.5 + calculator | 100% · 11/1174–100 | 0 | 3k tokens | $0.16 |
| DeepSeek V4.1 Flash + calculator | 100% · 11/1174–100 | 0 | 10k tokens | $0.01 |
11 questions, one from each family plus a trap whose honest answer is “I can’t answer that.” Ran out of room: used its whole reply budget thinking and never answered (scored as not right). Sonnet alone shows 10 because one reply was cancelled to stay inside the run’s spending cap. The shaded band behind each bar is the range the true score likely falls in, given so few questions.
One question, both ways
Ray and Linda want to start Roth conversions from Ray's traditional IRA this year and want to know how much to convert.
- The calculator’s best plan
- $118,000 a year for 8 years (2026–2033)
- DeepSeek V4.1 Flash, on its own
- Convert nothing (a projected $134,445 less after-tax wealth over 25 years; a simulation, not a real household)Wrong
- DeepSeek V4.1 Flash + calculator
- $118,000 a year × 8 yearsRight
Confident, wrong, and the projected after-tax wealth it gives up over 25 years (simulated households, not real ones)
- 1How much to convert to Roth, and for how many years (DeepSeek V4.1 Flash, on its own). With the calculator: right.−$134,445
- 2Converting inside the senior-deduction phase-out (DeepSeek V4.1 Flash, on its own). With the calculator: right.−$72,529
- 3How much to convert to Roth, and for how many years (Claude Haiku 5.5, on its own). With the calculator: right.−$26,002
- The calculator’s answers held. With the calculator, all 3 models made the right decision on every question, with every number within 2% of the key, and never went against it. They also thought far less: about 3k tokens for Sonnet instead of 37k.
- Hard on purpose. Code checks that no single tax rate or skipped rule lands within 2% of the answer. The answers are rechecked by separate programs (203 of 203 agree; Technical details).
- Still small. One try per question, three models. The intervals are wide on purpose: this shows the gap, not a ranking.
A separate result, not in the table above
GPT-6 Luna through Codex, without a calculator: it said it couldn’t
- Questions
- the same 11 households, trap included
- Model
- GPT-6 Luna through Codex (the model on my ChatGPT subscription, run in OpenAI’s Codex app), with no calculator, code or web
- Settings
- medium effort, no length limit, one try each; a different setup from the three models above
- Graded
- the same way: the right decision, and every number within 2%
Result. It declined all 10 questions that had an answer, typically like this: “A reliable answer requires a full multi-year projection of taxes, RMDs, premiums, and account balances.” (RMDs are the withdrawals the IRS requires from retirement accounts.) It also rightly declined the trap, so it scored 1 of 11.
Why this matters. That’s an honest answer, and it’s the case for a calculator. A real decision needs a 25-year projection. GPT-6 Luna declined every one; some other models answered some of these correctly without a calculator (Claude Sonnet 5.5 got 7 of 10), and with one, all three models tested got every decision right.
Short test: leave a fact out. Does it ask, or guess?
- Questions
- the same household questions (not the trap), each with the one deciding fact left out: a Social Security amount or an IRA balance. The file says something may be missing, not what.
- Model
- DeepSeek V4.1 Flash, on its own and with the calculator
- Settings
- as above; the model may ask a scripted client up to three questions before answering
- Graded
- the same way, against the full facts
Result. With the calculator, DeepSeek asked for the missing fact 9 times out of 10.
| Model | Fully right | Asked for the missing fact | Cost per question |
|---|---|---|---|
| DeepSeek V4.1 Flash, on its own | 40% · 2/512–77 | 5 of 5 | $0.01 |
| DeepSeek V4.1 Flash + calculator | 90% · 9/1060–98 | 9 of 10 | $0.01 |
One question per family (the trap has nothing to leave out). The alone run was stopped after 5 questions: answering without the calculator took 10–15 minutes each. Fully right is graded against the full facts, by the same grader as above. The shaded band behind each bar is the range the true score likely falls in, given so few questions.
What it asked (word for word)
- 1“Is Frank receiving Social Security now—and if so, what is his monthly benefit and at what age did he claim? If he is not yet claiming, what is his monthly benefit at full retirement age and when does he plan to claim?”
- 2“The options call for converting from Glen's traditional IRA, but the account list you gave me only has Maria's traditional IRA ($70,000) and Glen's Roth IRA — there's no Glen traditional IRA or 401(k) listed. …”
- 3“What is Ken's traditional IRA balance as of December 31, 2025, and is any of it after-tax money (i.e., does he have a Form 8606 basis)?”
Why guessing can’t pass
- Every tested guess fails. Before a question is used, the engine answers it 4–5 times with the missing fact guessed (none, half, 75%, 125%, 150%). All 69 of 69 questions grade wrong under every guess.
- The client never volunteers. It’s code, not a model: it answers only a question on the missing fact’s topic, and says nothing new to anything else.
- Checked with stand-ins. A stand-in that asks the right question gets 69 of 69 fully right; one that guesses gets 0 of 69; one that asks vague questions never gets the fact (0 of 69).
See it work
Watch the calls, or see it in a chat
- Replays: each household decision above, step by step: the household, every calculator call, and the answer against the key.
- Demo walkthrough: the same calculator as a private add-on for Claude and ChatGPT. Not open to the public; that page shows its real replies.
For the technical reader
The calculator is an MCP server (the Model Context Protocol, the standard way to plug tools into AI models). The benchmark runners and the chat add-on call the same code. The first test offered federal tax, taxable Social Security, required withdrawals, Social Security benefits, Medicare premiums and tax tables. The November set offered those plus a year-by-year scenario runner; the Roth-conversion solver and the chat front door were left out to keep the instructions short.
Calculator test 1: from calculator 1.0 (just the number) to 1.1 (the number plus the rule), the same questions both ways; exact McNemar test, p < 0.001. Shaded bands in every table are 95% intervals (Wilson).
Calculator test 2 (the 42 questions in the newer set; low effort; one run each way; graded by code within $2–5). Token budget: 4,000 output tokens per turn, up to 8 turns. 10 of 336 calculator replies used more than 4,000 in total, the longest 7,135.
DeepSeek V4.1 Flash with the calculator against Claude Sonnet 5.5 on its own, the same 42 questions, first run each: 10 only DeepSeek got right, 3 only Sonnet got right; exact McNemar p ≈ 0.0923. Higher observed score, not a significant difference. DeepSeek alone → with: 21 fixed, 0 broken, p < 0.001. Claude Haiku 5.5: 8 fixed, 2 broken, p = 0.1094.
First test: Claude Haiku 5.5 through Anthropic’s API. November set: every model through OpenRouter (a service that relays requests to many models), each pinned to one host. The calculator prices each choice; it never picks investments or products.