# PlannerBench: the whole site as one document > Can AI answer retirement questions correctly? PlannerBench is a set of retirement planning questions with known right answers, built by a financial advisor in Sun City West, Arizona. AI models are given the questions and graded on what they say. Every page of https://plannerbench.org in reading order, made from the finished pages at build time. Tables are kept as tables; each chart is followed by a table of the numbers it shows; text behind toggles is written out. The page map is at https://plannerbench.org/llms.txt. --- ## PlannerBench Source: https://plannerbench.org/ · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt **Can AI answer retirement questions correctly? I tested it.** PlannerBench is a set of retirement planning questions with known right answers. I give the questions to AI models and grade what they say. The questions cover taxes, Social Security, Medicare, required minimum distributions, and Roth conversions under 2026 rules. I built a planning engine to work out the correct answers, and a second, separately written program to double-check them. By Peyton Thalman, financial advisor in Sun City West, Arizona. ### Why I built it I’m a financial advisor, and I meet a lot of retirees by knocking doors. Many of them tell me they ask ChatGPT their money questions. When I ask how they know the answer is right, they say they Googled it and the top result agreed. That top result is usually Google’s AI. I wanted a real way to check, so I built one. [How I built it →](https://plannerbench.org/built.html) ### Results - **Questions with one rule:** The top models got [about 97%](https://plannerbench.org/results.html#one-rule) right. - **Questions where several rules interact:** This is where most models fall apart. On the [42 hardest questions](https://plannerbench.org/results.html#several-rules), GPT-6.1 Sol got [98%](https://plannerbench.org/results.html#several-rules), Claude Sonnet 5.5 got [62%](https://plannerbench.org/results.html#several-rules), and GPT-6 Luna got [17%](https://plannerbench.org/results.html#chatgpt).[*](https://plannerbench.org/#note-luna) Cheap and free models were all [under 30%](https://plannerbench.org/results.html#several-rules). - **With a calculator that shows its work:** When the calculator gave the rule and the IRS form line along with the number, Claude Haiku went from [40% to 100%](https://plannerbench.org/results.html#calculator). - **When the client’s information is wrong:** I put one impossible fact in each client file, like a birthday that makes the client 14. Most models gave advice anyway. GPT-6 Luna gave retirement advice to a 14-year-old. Even the best models missed [20–27%](https://plannerbench.org/results.html#wrong-facts) of these. [See all results →](https://plannerbench.org/results.html) ### If you use AI for retirement questions Check the ages, balances, and income it used. Ask which rule it’s applying. If it can’t tell you, or it got your facts wrong, don’t act on the answer. For decisions you can’t undo, like when to start Social Security, talk to a person. ### For financial firms Your clients are already asking AI before they ask you. Most can’t tell when it’s wrong, and the AI usually won’t tell them when the facts they gave it are wrong. [More for financial firms →](https://plannerbench.org/firms.html) ### For AI labs Every number answer here is exact and checked, so a script can grade it without a person or another AI deciding what’s right. That makes it useful for testing and training models on the money questions people are already asking. [More for AI labs →](https://plannerbench.org/labs.html) ### Where this is going I believe the best AI models will soon be able to do financial planning at the level of the best human advisors. The free models will catch up to today’s best paid ones. This needs to be done at a bigger scale than I can manage on my own, and I’d like to take it further as part of a team. [More on where I want to take this →](https://plannerbench.org/built.html#next) [See example questions →](https://plannerbench.org/questions.html) [How I built and checked it →](https://plannerbench.org/built.html) [peytonthalman.com](https://peytonthalman.com) *GPT-6 Luna is the model behind the regular ChatGPT app. I tested it through Codex with code and web access off. That’s close to the ChatGPT app, but not identical. Decision support, not advice. Answers follow IRS, SSA, and CMS publications. Nothing on this site is tax or investment advice. --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. --- ## The numbers behind each finding Source: https://plannerbench.org/results.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt Results Each section below backs up one line of the [home page](https://plannerbench.org/), with the short table and a link to the full detail. How the questions were built and checked, and how the tests differ, is on [Technical details](https://plannerbench.org/method.html). Finding 1 · First test · October 2026 ### One rule at a time, it’s great A **one-rule question** applies a single rule, like the tax on an income or this year’s required IRA withdrawal. | Model | One rule | Several rules at once | |---|---|---| | Claude Sonnet 5.5 (Anthropic) | 97% (89–99) | 37% (22–54) | | Claude Haiku 5.5 (Anthropic) | 97% (89–99) | 17% (7–34) | | DeepSeek V4.1 Flash (DeepSeek · free to download) | 86% (77–91) | 36% (23–50) | | GLM 5.3 Flash (Z.ai · free to download) | 68% (58–77) | 27% (16–41) | | gpt-oss-120b (OpenAI · free to download) | 36% (26–46) | 5% (2–18) | Claude Sonnet 5.5 got 97% of one-rule questions right, and Claude Haiku 5.5 got 97%. When several rules met in one household, Sonnet dropped to 37%. The shaded band behind each bar is the range the true score likely sits in. [The first test in full →](https://plannerbench.org/alone.html#october) Finding 2 · Newer, harder set · November 2026 ### Several rules at once, it depends heavily on which AI 42 harder questions, each making two or more rules meet: the largest Roth conversion that won’t raise Medicare premiums two years later, a widow’s first year of taxes, Arizona state tax for retirees. 11 models, at low reasoning effort with a 4,000-token reply limit. | Model | Right, of the newer set | |---|---| | GPT-6.1 Sol (OpenAI · top-priced) | 98% · 41/42 (88–100) | | GPT-6.1 Terra (OpenAI · mid-priced) | 80% · 33.5/42 (65–89) | | Claude Sonnet 5.5 (Anthropic · top-priced) | 62% · 26/42 (47–75) | | DeepSeek V4.1 Flash (DeepSeek · cheap and free) | 29% · 12/42 (17–44) | | Llama 4 Maverick (Meta · cheap and free) | 6% · 2.5/42 (2–17) | GPT-6.1 Sol got 98%, GPT-6.1 Terra 80%, Claude Sonnet 5.5 62%, and all 5 cheap and free models under 30%. These settings differ from the October test, so don’t compare the two sets of scores. [All 11 models in full →](https://plannerbench.org/alone.html#november) Finding 2 · The ChatGPT model · November 2026 ### GPT-6 Luna GPT-6 Luna, asked the same 42 hard questions through OpenAI’s Codex app on my ChatGPT subscription, with code and web off, graded by the same answer key. | Questions | GPT-6 Luna | |---|---| | First test · one rule | 71% · 42 of 59 | | Newer, harder set | 17% · 7 of 42 | | Wrong facts · caught the planted mistake | 50% · 15 of 30 | It ran with more time to think and no length limit, in a different setup from the models above, so it isn’t ranked with them. Why, and how the run was done, is on [Technical details](https://plannerbench.org/method.html#chatgpt). [The GPT-6 Luna run in full →](https://plannerbench.org/alone.html#chatgpt) Finding 2 · Calculator test · October 2026 ### A calculator that only gave the number didn’t fix it. One that showed the rule did. Claude Haiku 5.5 on the 50 hardest first-test questions, asked three ways: on its own; with a calculator that sends back only the number; and with one that also shows the rule and the IRS form line behind it. **Result.** 40% on its own, 62% with only the number, 100% with the rule shown. On 50 held-back questions nobody tuned against: 44% → 96%. | Claude Haiku 5.5 | The hardest questions | Held-back questions | Dollars off per question, per year | |---|---|---|---| | On its own | 40% · 20/50 (28–54) | 44% · 22/50 (31–58) | $318 (held-back $216) | | Calculator 1.0: just the number | 62% · 31/50 (48–74) | (not asked) | $229 | | Calculator 1.1: the number, the rule and its source | 100% · 50/50 (93–100) | 96% · 48/50 (87–99) | $0 (held-back $4) | The same 50 questions in every row (5 tasks, 10 each). The right number alone fixed 11 questions it had wrong and broke 0 it had right; adding the rule and its source fixed 19 more and broke 0. With the same questions asked both ways, a jump that size is very unlikely to be luck (the statistics are under “For the technical reader” below). On 50 held-back questions, never seen while the calculator was built: 26 fixed, 0 broken. The shaded band behind each bar is the range the true score likely falls in, given so few questions. **Not every model was rescued.** On the newer set, DeepSeek V4.1 Flash rose from 29% to 79% with the calculator; three other cheap models stayed at 10% or less. [All three calculator tests in full →](https://plannerbench.org/calculator.html) Finding 3 · Wrong facts · November 2026 ### It believes whatever you tell it 30 client files, each with one planted mistake (two digits swapped in a balance, a birth year that makes a retiree 14, monthly spending far beyond what the family owns). A model **caught** it only if it named the mistake *and* stopped to ask instead of giving a number. | Of 30 planted mistakes | Recognized it | Stopped and asked | Caught (both) | Refused a fine file | |---|---|---|---|---| | GPT-6.1 Sol (top-priced) | 100% | 80% | **80%** | 3% | | Claude Sonnet 5.5 (top-priced) | 93% | 73% | **73%** | 0% | | Gemini 3.8 Flash (mid-priced) | 92% | 100% | **92%** | 50% | | DeepSeek V4.1 Flash (cheap and free) | 76% | 68% | **61%** | 0% | | gpt-oss-120b (cheap and free) | 3% | 7% | **3%** | 5% | Even the best at catching mistakes without false alarms, GPT-6.1 Sol and Claude Sonnet 5.5, missed 20–27%. Three of the cheap and free models (Mistral Small, gpt-oss-120b and Llama 4 Maverick) caught 1–3%. An AI judge decided whether a reply named the mistake; how that was checked is on [Technical details](https://plannerbench.org/method.html#ai-work). One example A client file said a retired school administrator with a $785,000 IRA was born in 2012, and asked how much she must withdraw this year. One model (gpt-oss-120b) replied: > “Priya will be 14 years old throughout 2026 … no required minimum distribution is required. ANSWER: 0” It saw that she was 14 and kept going. With her real birth year, the answer is $34,279. [The wrong-facts test in full →](https://plannerbench.org/wrong-facts.html) [Next **How I built it**](https://plannerbench.org/built.html) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## How I built PlannerBench Source: https://plannerbench.org/built.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt I’m a financial advisor, not a programmer. I built all of this by telling AI coding tools what to build (mostly Claude Code) and checking what they made. A lot of it happened at 3:45 in the morning before the gym, on the stair stepper, or while I pushed my daughter in a stroller and worked on my laptop remotely from my phone. Running the latest full round of tests cost $4.51. ### It started as a different project I wanted to see if I could build my own version of the planning software advisors use, and fill it in by talking instead of typing everything in by hand. Getting the retirement math right turned out to be hard. Taxes, Social Security, and Medicare all affect each other, and making my version correct took a lot of work. That made me wonder how the AI models do on this math on their own. In my testing they made a lot of mistakes. ### Then it got personal I meet a lot of my prospects by knocking doors. Many of them already have an advisor and ask ChatGPT their money questions anyway. When I ask how they know the answer is right, they say it seems right. They Google it, and the top result agrees, but that top result is usually Google’s AI. One prospect I’d been following up with told me they had a question about a 529 versus a UTMA account for their kid. They felt it was too dumb a question to ask me, so they asked ChatGPT. That’s fine for a simple question. On more complicated stuff, I don’t think people realize what risks they’re taking. So I wanted an actual answer: how do you check whether the AI is right? ### The parts, in plain English 1. **The calculator (planning engine).** A program that knows the 2026 rules for federal taxes, Social Security, Medicare premiums, and required withdrawals from retirement accounts. Give it a household and it works out the exact numbers. Think of a very focused TurboTax for retirement. 2. **The answer key.** Thousands of retirement questions, each with one exact answer, usually a dollar amount, worked out by the calculator. 3. **Checking the checker.** If the calculator is wrong, the whole test is wrong. So I had a separate AI write a second calculator from the written rules only, without ever seeing the first one’s code, in a different programming language. On 4,000 made-up households, the two have to agree within $1. The first time I compared them, they disagreed on 3 households, and that’s how I found a real bug in my calculator. Free outside tax software called PolicyEngine-US also rechecks the tax answers. When anything disagrees, the IRS, Social Security, or Medicare publication decides. A vote between AIs doesn’t. 4. **The test.** I gave the same questions to 11 AI models from 7 companies, plus GPT-6 Luna. A script grades the number answers, so nobody’s opinion decides what’s right, mine included. 5. **Secret questions.** I kept some questions locked away and never used them while I was building. It’s like not letting a student see the final exam. If a fix works on questions I never looked at, it’s a real fix and not me accidentally teaching to the test. 6. **Giving the AI the calculator.** My first idea was to let the AI use my calculator. The surprise was that it often ignored the right number and argued with it. What fixed it was changing what the calculator handed back. Along with the number, it now shows the rule it used and the IRS form line. After that, Claude Haiku went from 40% to 100% on 50 hard questions, and 96% on the 50 secret ones. 7. **The wrong-facts test.** At the doors, people give wrong information all the time: the wrong birthday, transposed digits, a guess at what they spend. So I made 30 client files that each had one impossible fact (like a birthday that makes a retiree 14) and 40 normal files. The question was whether the AI notices and asks, or just gives advice anyway. An AI graded these replies, and an AI from a different company re-graded 176 of them and agreed 97.7% of the time. 8. **Having another AI audit everything.** Since AI built most of this, I had OpenAI’s top model review the whole project, looking for mistakes and claims that went too far. It found 12 problems. I fixed all of them and published what they were. ### What I learned The models mostly fail by skipping steps. They assume things instead of asking, and they lean on what they remember from training instead of the current rules. When you tell them what they need to ask and what to think about before they answer, they do much better. Even cheap models get pretty close. The expensive models do great on their own. GPT-6.1 Sol scored 98% on the hardest questions. But it sits behind a $20 a month subscription, and paying per question gets expensive fast across thousands of advisors and their clients. GPT-6 Luna, the cheaper model behind the regular ChatGPT app, scored 17% on its own. So for most people, the answer isn’t a smarter model. It’s asking the right questions in a set order, giving the model a calculator that shows its work, and having it flag things that don’t make sense. ### Where I want to take this I believe the best AI models will soon be able to do financial planning at the level of the best human advisors. People are already asking AI for advice, and before long it will be hard to tell the best AI advice apart from the best human advice. Right now the free and cheap models aren’t great at this, but the best models already do very well, at least on my small test. Eventually the free models will be as good as today’s best paid ones, and the best paid models will be at a level that’s hard to picture. That’s why this matters now. Checking whether AI gets retirement math right, and making it better at it, needs to happen at a much bigger scale than one guy can do on his own. I’d like to take PlannerBench further as part of a team, with real resources behind it. ### The limits This is one person on a small budget, testing for a few dollars a round. The samples are small. No CPA has signed off on the answer key yet. An AI, not a person, graded the wrong-facts test. I tested the ChatGPT model through OpenAI’s Codex app with code and web access off, which is close to the ChatGPT app but not the same. All the details are in [Technical details](https://plannerbench.org/method.html). ### Who did what I picked the problem, decided what counts as a correct answer, set the rules, and approved every test and every dollar spent. AI coding tools did the building and ran the tests. Separate AIs, outside tax software, and another company’s model checked the work. [For AI labs →](https://plannerbench.org/labs.html) [For financial firms →](https://plannerbench.org/firms.html) [Technical details →](https://plannerbench.org/method.html) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## Technical details Source: https://plannerbench.org/method.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt Technical details The plain answer The planning engine (the program that works out the right answers) and a second, separately written program both compute every answer in the first test, and they must agree within $2. Outside tax software built by other people recomputes the tax answers. Another engine, written by a separate AI agent that was never allowed to see the planning engine’s code, agrees within $1 on every one of 4,000 random households, and every question in the newer set must match it. Two human checks are still owed: a CPA’s sign-off and a person spot-checking the AI judge in the wrong-facts test. That’s the whole idea. The rest of this page is the detail: [every test](https://plannerbench.org/method.html#tests), [the ChatGPT model](https://plannerbench.org/method.html#chatgpt), [the checks](https://plannerbench.org/method.html#checks), [the statistics](https://plannerbench.org/method.html#proof-tech), [the choices and mistakes behind the build](https://plannerbench.org/method.html#build-log), and [the words used on this site and the limits](https://plannerbench.org/method.html#glossary). How I built it, in plain English, is on [How I built it](https://plannerbench.org/built.html). ### Every test October and November used different settings, so compare within a test, not across. October: high reasoning effort, long replies. November: low effort, replies up to 4,000 tokens. | Test | When | What it asked | How many | Which AIs | |---|---|---|---|---| | [First test](https://plannerbench.org/alone.html#october) | Oct | One-rule questions, then several rules at once | 140–210 each, of 2,800 | six models | | [Newer, harder set](https://plannerbench.org/alone.html#november) | Nov | Harder questions where several rules meet | 42 | 11 models | | [GPT-6 Luna through Codex](https://plannerbench.org/alone.html#chatgpt) | Nov | The same questions, through my ChatGPT subscription | 42 + 139 from the first test | GPT-6 Luna | | [Calculator test 1: number vs rule](https://plannerbench.org/calculator.html#tools) | Oct | Does a calculator help if it shows only the number, or the rule too? | 50 + 50 held back | Claude Haiku 5.5 | | [Calculator test 2: who it rescues](https://plannerbench.org/calculator.html#cheap) | Nov | Which models the calculator lifts | 42 | 8 models | | [Calculator test 3: whole-household decisions](https://plannerbench.org/calculator.html#gauntlet) | Oct | Real choices over 25 years, like when to claim Social Security | 11 | 3 models | | [Wrong facts](https://plannerbench.org/wrong-facts.html) | Nov | Client files with one fact that can’t be right | 30 + 40 that were fine | 11 models + GPT-6 Luna | All in 2026. Price groups, by what each model costs to use: **top-priced:** GPT-6.1 Sol, Claude Sonnet 5.5, Gemini 3.1 Pro · **mid-priced:** GPT-6.1 Terra, Gemini 3.8 Flash, Claude Haiku 5.5 · **cheap and free:** DeepSeek V4.1 Flash, GLM 5.3 Flash, gpt-oss-120b, Llama 4 Maverick, Mistral Small. ### About ChatGPT: GPT-6 Luna through Codex The regular ChatGPT app uses GPT-6 Luna, a different, cheaper model than GPT-6.1 Sol (the one that got 98% of the hardest questions). I ran Luna through OpenAI’s Codex app (a ChatGPT-subscription tool for running tasks) with code and web turned off, in small batches, so it’s close to, but not the same as, typing into the ChatGPT window. It also ran with more time to think and no length limit, so its 17% on the hardest questions isn’t ranked with the other models. How the run was done, and the first attempt that was thrown out, is on [AI on its own](https://plannerbench.org/alone.html#chatgpt). ### The answer key, in plain words Every number question has one exact answer, usually a dollar amount. The planning engine, a program I directed AI agents to build, computes it from the 2026 rules: IRS forms and publications, the Social Security Administration’s operating manual, and Medicare’s premium tables. For those number questions a script does the grading, so nobody’s opinion is involved, including mine. The wrong-facts test is the exception: whether a reply flagged a planted mistake takes judgment, so an AI judge decides, checked as described below. When AI models disagree with the key, the published source decides, not the models. In the first test, models from several companies agreed with each other and against the key on the new senior deduction. The IRS form, and separately PolicyEngine, showed the key was right. ### The checks, with their scores Each card says what was checked and how far the check reaches. **4,000 / 4,000** Another engine, written without seeing the first A separate AI agent wrote another engine, in a different programming language, from the written rules alone, and was never allowed to read the first. Both run on random households; every number for every year must agree within $1. Its first run found a real bug in the main engine. Every question in the newer, harder set also has to match it before it can be used. **1,600 / 1,600** Outside tax code PolicyEngine-US, open-source tax software written by other people, recomputes every tax answer in the first test, all within $2, the new senior deduction included; and 400 / 400 on the held-back questions (private ones never published). It doesn't model required withdrawals, Social Security claiming or Medicare premiums, so those rest on the published tables and the second engine. **203 / 203** Whole-household decisions, recomputed Every option in every public whole-household question is rerun by the second engine, and its 2026 federal tax return by PolicyEngine, before the question can be used. **2,750 / 2,750** The calculator can answer everything Before any model used the calculator, a script checked that calling it correctly reproduces every number answer in the first test: the 2,200 public questions that have a number, plus 550 held-back ones. The other 600 of the 2,800 public questions are compliance questions, answered with sentences or flags rather than a number, so there is nothing for a calculator to compute. ### What checked the AI’s work - **The engines above.** A second engine written by a separate agent that never read the first, and outside tax software written by people, recompute the answers. - **A second judge from another lab.** In the wrong-facts test, an AI judge (Claude Haiku 5.5) reads each reply and records whether it flagged the problem. A judge from OpenAI (GPT-6 Luna) re-read 176 of those replies and agreed on 97.7%. Where a reply plainly named the problem but the judge marked it as missed, the lead AI agent corrected the label (18 replies). Two AIs agreeing, or one AI correcting another, is not the same as a person checking. - **The GPT-6 Luna run.** GPT-6 Luna through Codex (the model on my ChatGPT subscription): its replies were graded by the same answer keys and the same scripts as every other model. On wrong facts the same judge (Claude Haiku 5.5) used the same instructions, and the lead AI agent read every reply the judge marked as not catching the mistake (12 replies; 0 labels changed). The replies the judge marked as caught haven’t been re-read yet. A first attempt was thrown out because most replies were a bare final number with no reasoning, which isn’t how ChatGPT answers a person; the details are on [AI on its own](https://plannerbench.org/alone.html#chatgpt). - **An outside audit.** On November 15, 2026, OpenAI’s GPT-6.1 Sol read the whole project, without changing anything, and looked for errors. It found 12, all fixed on this version; the list is in the [report](https://plannerbench.org/report.html). - **Still owed.** A CPA (a licensed tax accountant) hasn’t signed off on the answer key yet; worked solutions are ready for one to check with a calculator. That would be a human check on top of the independent programs, which already agree. And no person has yet spot-checked the AI judge’s labels in the wrong-facts test. ### What every first-test question must pass 1. **Published examples.** Cases whose answers come straight from IRS, Social Security and Medicare tables must match exactly. 2. **Two independent programs.** The planning engine and a second, separately written program compute every answer and must agree within $2. They agree on all 2,800 questions in the first test. 3. **Rules that must always hold.** Thousands of random households test things like “more income never lowers tax” and “waiting a month never lowers a Social Security check.” 4. **Dataset health.** No duplicates, and no task where one answer is common enough to guess. 5. **An outside tax model, where it reaches.** PolicyEngine-US recomputes the first test’s tax answers and the 2026 federal returns in the whole-household decisions. It doesn’t cover required withdrawals, Social Security claiming or Medicare premiums. The newer set’s questions must match the second engine before they’re used; its Arizona state-tax questions have no outside check. The human checks above are still open, so these are the automatic checks, not a sign-off. **For the technical reader: statistics, scripts and where each check lives** - **Every accuracy carries a 95% interval:** the range the true score likely sits in. Small runs show wide ranges on purpose. - **Comparisons are paired.** “Alone vs. with the calculator” asks the same model the same questions and counts how many flipped each way, then uses McNemar’s test (a standard check of whether that many one-way flips could be luck). - **Failures are their own bucket.** A reply that ran out of tokens is published and counted as not right, never quietly dropped. - **Mistakes are priced.** Each wrong number in the first test is also scored in dollars a year off, so a ten-cent miss and a large one don’t count the same. - **Hosts are pinned.** Every outside model runs on one named host with data collection off; the private set never goes to free endpoints. - **A held-back set.** Held-back questions are private questions, made from a secret seed (a starting number for the generator), that pass the same checks and are never published, so a model that trained on the public set still meets new ones, and nothing is tuned on them. - **What rebuilds the newer set.** `v2/grade.py` regrades it from saved replies; `v2/build_hard.py` and `v2/build_bad.py` rebuild its questions and stop if the engine and an independent check disagree. - **Same model, two routes.** Claude Haiku 5.5 answered the same 140 questions directly and through OpenRouter (a service that relays requests to many models): 77% and 77%. No difference detected in this check, which is not proof the route never matters. (checks/HARNESS.md) - **One script rebuilds the first test.** reproduce.sh (a script anyone can run from the code) re-checks the first test’s answer key, regrades every saved first-test reply, rebuilds its scores and the calculator runs on its 50 hardest questions, and compares them with the published copy; any difference fails it. It doesn’t regenerate the questions or cover the newer set. - **Where each check’s record lives.** Another engine, written without seeing the first: engine repo: verify/COMPARE.md (rules v1.4.1, 2026-10-23) · Outside tax code: checks/POLICYENGINE.md · Whole-household decisions, recomputed: checks/GAUNTLET.md · The calculator can answer everything: checks/tools_check.py (README log, 2026-10-21). (“engine repo” is the planning engine’s own code repository; verify/COMPARE.md there is the two engines’ comparison report.) ### Behind the build The main choices I made, the mistakes that got caught, and when each piece was built. The plain-English version is on [How I built it](https://plannerbench.org/built.html). **The main choices I made, and why** - **When the AI ignored the calculator, I changed what the calculator says back.** I could have written a stronger prompt or used a bigger model. Instead the calculator now shows the rule and the IRS form line next to the number, which fixes the cause. I tested that on one model; how it carried over to others is on [With a calculator](https://plannerbench.org/calculator.html). - **A few models, cheapest first, inside small budgets.** October: two Claude models plus four low-cost models from other companies. November: 11 models from seven companies, plus the ChatGPT model tested separately. The goal was to show the gap and why it happens, not to rank every model. - **“Better plan” means more money left after taxes at the end.** Not lowest tax or highest income. That’s what the family actually keeps. - **When two plans are close, say so.** If two plans end within a rounding error of each other over 25 years, picking a “winner” would be false precision. - **The chat version keeps nothing people type.** People enter real finances, so nothing is saved. The cost is no usage numbers, so quality is measured with made-up clients instead. **Mistakes the AI made while building, and what caught them** - **A bug in the calculator.** The second calculator disagreed on 3 of 2,000 households. One was a real error in mine: a 2027 tax threshold. After the fix, the two agreed on all 4,000 households in the next comparison. - **A Medicare premium off by one level.** Income exactly at a cutoff was put one level too low. A check found it and it was fixed. - **Unfair questions.** Reading the first wrong-facts replies, a model rightly asked about a fact the file never gave. 14 files were unclear. I fixed them, asked again, and left the old replies out of the scores. - **A spending cap that leaked.** One run went $0.39 over its cap because requests already in progress weren’t counted. The cap now sets aside the most each request could cost. - **The first ChatGPT-model run cut corners.** Asked to do dozens of questions at once, it wrote bare final numbers instead of answering each one like a chat would. I threw that run out and redid it in small batches. Lesson: when an AI does work in bulk, check how it did the work, not only the score. - **Grading and wording errors.** The outside audit found replies that ran out of room counted the wrong way, some judge labels that missed a model naming the mistake, and sentences that claimed more than the data showed. All fixed; the [report](https://plannerbench.org/report.html) lists each one. On the wrong-facts test the lead AI also corrected 18 of the judge’s labels, every change listed; a person still needs to spot-check them. **Timeline** - **Oct 15:** the calculator (federal tax, Social Security, required withdrawals, Medicare premiums), checked for rule errors the same day. - **Oct 20:** the first test: questions with a provable answer key, outside tax software as a referee, and the first models graded. - **Oct 21:** the calculator as a tool AI can use. Version 1.0 gave only the number and got ignored; 1.1 showed the rule and the form line, and the AI mostly stopped arguing. Also the second calculator, written from the rules alone. - **Oct 22:** whole-household decisions over 25 years, and the calculator inside Claude and ChatGPT, tested with made-up clients first. - **Nov 14:** the harder questions and the wrong-facts test, after I saw the Claude models had nearly maxed out the simple questions and every fact I’d given them was true. - **Nov 15:** an outside audit by OpenAI’s GPT-6.1 Sol. It found 12 errors, all fixed, and the site was rewritten to read simply. ### Words on this site - **One rule**: One tax or retirement rule, like this year’s tax brackets or a required IRA withdrawal. - **Several rules at once**: Taxes, Social Security and Medicare affecting each other, the way they do in a real household. - **The planning engine**: The program I had built for the 2026 IRS, Social Security and Medicare rules. It computes the answer key, and it is the calculator the AI models could call. - **Best**: On scores, right most often. On wrong facts, catching the most mistakes without refusing files that were fine. - **Caught**: On wrong facts, the reply recognized the planted mistake *and* stopped to ask, with no number. - **AI judge**: Another AI model reading each wrong-facts reply to decide whether it named the mistake. No person has checked its labels yet. - **Held-back questions**: Secret questions never used while building, kept only to check that a fix is real. - **CPA**: A licensed tax accountant. A CPA’s sign-off on the answer key is still to come. - **PolicyEngine-US**: Outside tax software built by other people. It rechecks the first test’s tax answers. - **Price groups**: By what each model costs to use: top-priced, mid-priced, and cheap and free (listed under [Every test](https://plannerbench.org/method.html#tests)). **The limits.** Small samples; on the wrong-facts test, an AI judge decided whether a model named the mistake; no CPA sign-off yet. [Back to the start **PlannerBench in one page**](https://plannerbench.org/) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## For AI labs Source: https://plannerbench.org/labs.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt **What this is:** A test set of retirement planning questions where every answer is exact and checked. The answers come from a calculator built on 2026 IRS, Social Security, and Medicare rules, confirmed by a second calculator written separately and by outside tax software. This is an area where real people are already asking AI for advice and acting on it. ### What it shows - The top models are nearly maxed out. GPT-6.1 Sol scored 98% on the hardest set. - Cheap models aren’t. They scored under 30%, and GPT-6 Luna scored 17%. - None of them reliably catch wrong facts. Even the best models missed 20–27% of planted errors and gave advice anyway. ### Why it’s useful for training Every number answer can be graded by a script, so there’s no human or AI judge in the loop for those. I think a cheap model could close most of the gap quickly with reinforcement learning (training it by rewarding correct answers). The wrong-facts files add a second thing to train: knowing when to stop and ask instead of answering. ### What I’d build next Subtler, more realistic wrong facts. A CPA sign-off on the answer key. A person checking the AI judge’s labels. [How I built it →](https://plannerbench.org/built.html) [Technical details →](https://plannerbench.org/method.html) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## For financial firms Source: https://plannerbench.org/firms.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt **Your clients are already asking AI.** Even in a retirement community where most people have an advisor, prospects tell me they ask ChatGPT first. One told me a 529 question felt too dumb to ask me. They trust it because it seems right and Google’s AI agrees. ### What I found - On single rules, good models are almost always right. - On real retirement questions, where taxes, Social Security, and Medicare interact, it depends heavily on the model. The cheap and free models people actually use scored under 30%. - When a client gets their own facts wrong, most models give confident advice anyway. ### What fixes it, without paying for the most expensive model - Ask the questions that matter in a set order, so nothing gets assumed. - Give the AI a calculator that shows the rule behind every number. - Have it flag facts that don’t make sense before it answers. ### What a firm could do with this - **Know which questions need an advisor.** The test data shows where AI does well and where it breaks. - **Offer a “check your AI answer” tool on your website.** Someone brings the question they asked ChatGPT. The tool collects the facts that matter, flags what needs a professional, and asks if they’d like an advisor to reach out. People are using AI either way, so the firm might as well offer the version that checks itself. - **Keep costs down.** Structure and checks let cheap models do the job, which matters across thousands of advisors and their clients. [How I built it →](https://plannerbench.org/built.html) [Technical details →](https://plannerbench.org/method.html) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## AI gets one rule right. When rules meet, it depends which AI you ask. Source: https://plannerbench.org/alone.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt AI on its own · findings 1 and 2 The plain answer On the one-rule questions I tested, Claude Sonnet 5.5 and Claude Haiku 5.5 were right about 97% of the time. On questions where several rules meet, the way they do in a real household, scores spread out: in my newer, harder set, GPT-6.1 Sol got 98% right, GPT-6.1 Terra 80%, and all 5 cheap and free models scored under 30%. GPT-6 Luna through Codex (the model on my ChatGPT subscription), tested separately, got 17%. In the first test, many models made the same mistake on a tax rule that is new since 2025. ### One question, and where it goes wrong A real question from the first test (October 2026). 4 models from 3 labs gave the same wrong answer to the same question: $15,928. The list below has five rows with that number because I asked Claude Haiku 5.5 two separate ways (“second way” below; details under “How the first test was run”), and it made the mistake both times. The IRS form shows why. The question In 2026, a married couple filing jointly (both spouses age 65 or older) has $105,500 of traditional IRA distributions, $0 of long-term capital gains and qualified dividends, and $72,300 of Social Security benefits, with no wages. They take the standard deduction and have no credits. What is their total federal income tax including any Net Investment Income Tax, rounded to the nearest dollar? - **Answer key**: $16,152 - **Claude Haiku 5.5**: $15,928 (Wrong) - **Claude Sonnet 5.5**: $15,928 (Wrong) - **Claude Haiku 5.5 (second way)**: $15,928 (Wrong) - **DeepSeek V4.1 Flash**: $15,928 (Wrong) - **gpt-oss-120b**: $13,511 (Wrong) - **Qwen3.8 Flash**: No answer - **GLM 5.3 Flash**: $15,928 (Wrong) - **Claude Haiku 5.5, given the rule**: $16,152 (Right) - **Claude Sonnet 5.5, given the rule**: $16,152 (Right) IRS Schedule 1-A, Part V (2025 form) · Enhanced deduction for seniors A 2025 tax law gave each person 65 or older an extra $6,000 deduction (an amount taken off income before tax). It shrinks by 6% of income (the form’s “modified adjusted gross income”: roughly your total income before the standard deduction) over $150,000 on a joint return. Below is the 2025 version of the form, line by line; the question is about 2026, and the 2026 form runs the same steps on renumbered lines. - 31: Modified adjusted gross income — $166,955.00 - 32: Threshold, married filing jointly — $150,000.00 - 33: Line 31 minus line 32 — $16,955.00 - 34: Line 33 × 6% — $1,017.30 - 35: $6,000 minus line 34 — $4,982.70 - 36a: You, born before January 2, 1961: line 35 — $4,982.70 - 36b: Your spouse, born before January 2, 1961: line 35 — $4,982.70 - 37: Enhanced deduction for seniors — $9,965.40 What the AI models did They took the 6% once off the couple’s combined $12,000: $12,000 − $1,017.30 = $10,982.70. That’s $1,017.30 too much deduction, so the tax came out at $15,928 instead of $16,152. The form takes the 6% off each spouse’s $6,000 (lines 35, 36a and 36b). **How common was this mistake?** Across the two runs of the first test, 99 of the 342 wrong answers (29%) were this exact misreading. Most wrong answers matched no known misreading at all; of the 139 we could trace to a named one, it explains 71%. For Claude Sonnet 5.5 it was 19 of its 23 wrong answers. **Show it the rule, and it gets it right.** When I pasted the IRS’s instructions for the new deduction in front of the same questions, Claude Sonnet 5.5 went from 10% to 100% on the senior-deduction questions, and Claude Haiku 5.5 from 0% to 90%. On these questions, not knowing the new rule was the gap. First test · October 2026 ### The first test: simple questions almost always right, stacked ones mostly wrong A **one-rule question** applies a single rule, like the tax on an income or this year’s required IRA withdrawal. A **several-rules question** makes rules meet in one household, the way taxes, Social Security and Medicare do in real life. **Questions per model:** 140–210 of the 2,800 generated (10–15 per task) · **Models:** six from five companies (seven rows, because Claude Haiku 5.5 was asked two ways) · **Settings:** high reasoning effort, long replies · **Graded:** by a script against the answer key, within a dollar or two | Model | One rule | Several rules at once | |---|---|---| | Claude Sonnet 5.5 (Anthropic) | 97% (89–99) | 37% (22–54) | | Claude Haiku 5.5 (Anthropic) | 97% (89–99) | 17% (7–34) | | DeepSeek V4.1 Flash (DeepSeek · free to download) | 86% (77–91) | 36% (23–50) | | GLM 5.3 Flash (Z.ai · free to download) | 68% (58–77) | 27% (16–41) | | gpt-oss-120b (OpenAI · free to download) | 36% (26–46) | 5% (2–18) | Claude Sonnet 5.5 got 97% of one-rule questions right, and Claude Haiku 5.5 got 97%. When several rules met in one household, Sonnet dropped to 37% and Haiku to 17%. The shaded band behind each bar is the range the true score likely sits in, since each model answered only 10–15 questions per task. #### What the wrong answers were Each wrong answer checked against known ways of misreading the rules. The senior-deduction misreading is the biggest named cause; most other wrong answers matched no known misreading (“Unexplained”). One cheap model, gpt-oss-120b, used the required-withdrawal table the IRS replaced in 2022. (RMD, required minimum distribution: the amount you must take out of a traditional IRA each year once you reach your seventies.) **Chart: Wrong answers by type** | Item | Value | |---|---| | Claude Haiku 5.5 | 20 × senior deduction misread | | Claude Haiku 5.5 | 12 × unexplained | | Claude Sonnet 5.5 | 19 × senior deduction misread | | Claude Sonnet 5.5 | 4 × unexplained | | Claude Haiku 5.5 (second way) | 32 × senior deduction misread | | Claude Haiku 5.5 (second way) | 4 × other named mistake | | Claude Haiku 5.5 (second way) | 21 × unexplained | | DeepSeek V4.1 Flash | 21 × senior deduction misread | | DeepSeek V4.1 Flash | 6 × other named mistake | | DeepSeek V4.1 Flash | 20 × unexplained | | gpt-oss-120b | 8 × pre-2022 RMD table | | gpt-oss-120b | 5 × other named mistake | | gpt-oss-120b | 85 × unexplained | | Qwen3.8 Flash | 3 × pre-2022 RMD table | | Qwen3.8 Flash | 4 × other named mistake | | Qwen3.8 Flash | 5 × unexplained | | GLM 5.3 Flash | 7 × senior deduction misread | | GLM 5.3 Flash | 10 × other named mistake | | GLM 5.3 Flash | 56 × unexplained | **All models and columns (for the detail-minded)** Every model setup, every kind of question. “Compliance” questions ask a reviewer to catch a planted problem. In this test they were too easy: each question defined the issue it was about, so all seven setups scored 91–100%. The next version names issues without defining them. “Never finished” means the model used its whole reply budget thinking and never answered; that counts as not right. “A few steps” questions chain two or three calculations. “Overall” is every kind of question together. “Cost per 1,000 right” is what I paid in usage fees for every 1,000 right answers. | Model | One rule | A few steps | Several rules at once | Compliance | Overall | Never finished | Cost per 1,000 right | |---|---|---|---|---|---|---|---| | Claude Sonnet 5.5 (Anthropic) | 97% (89–99) | 90% (70–97) | 37% (22–54) | 100% (89–100) | 84% (77–89) | — | $13.27 | | Claude Haiku 5.5 (Anthropic) | 97% (89–99) | 75% (53–89) | 17% (7–34) | 100% (89–100) | 77% (70–83) | — | $1.29 | | DeepSeek V4.1 Flash (DeepSeek · free to download) | 86% (77–91) | 70% (52–83) | 36% (23–50) | 96% (85–99) | 75% (68–80) | 2% | $5.51 | | Claude Haiku 5.5 (second way) (Anthropic) | 92% (85–96) | 63% (46–78) | 20% (11–34) | 93% (82–98) | 73% (66–78) | — | $1.34 | | GLM 5.3 Flash (Z.ai · free to download) | 68% (58–77) | 50% (33–67) | 27% (16–41) | 93% (82–98) | 62% (55–68) | 3% | $3.74 | | Qwen3.8 Flash (Alibaba) | 43% (34–54) | 3% (1–17) | 0% (0–18) | 91% (79–96) | 45% (37–52) | 49% | $20.23 | | gpt-oss-120b (OpenAI · free to download) | 36% (26–46) | 10% (3–26) | 5% (2–18) | 91% (79–96) | 39% (32–45) | 13% | $6.06 | #### Give it the rule The same questions again, with the IRS’s two-line instructions for the new rule pasted in front. **Chart: Accuracy without and with the rule** | Item | Value | |---|---| | Haiku 5.5 · senior phaseout | 0% → 90% | | Haiku 5.5 · widow all in | 20% → 60% | | Sonnet 5.5 · senior phaseout | 10% → 100% | | Sonnet 5.5 · widow all in | 20% → 100% | Without the rule With the rule #### What the mistakes would cost a client A right-or-wrong score treats a Medicare premium that is ten cents off the same as a required withdrawal that is thousands of dollars too high. So here each answer is also scored by how many dollars a year it is off. A miss of $1,000 a year or more counts as a big one. | Model | Dollars off per question, per year | Big misses ($1,000+) | Typical miss | Worst miss | Questions scored | |---|---|---|---|---|---| | Claude Sonnet 5.5 (Anthropic) | $161 ($91–$253) | 4% (1–9) | $677 | $3,472 | 110 | | DeepSeek V4.1 Flash (DeepSeek · free to download) | $161 ($79–$294) | 3% (1–6) | $300 | $8,700 | 159 | | Claude Haiku 5.5 (Anthropic) | $171 ($109–$238) | 4% (1–9) | $568 | $1,512 | 110 | | Claude Haiku 5.5 (second way) (Anthropic) | $244 ($141–$376) | 4% (2–9) | $495 | $8,700 | 165 | | GLM 5.3 Flash (Z.ai · free to download) | $588 ($315–$904) | 11% (7–17) | $472 | $13,000 | 158 | | Qwen3.8 Flash (Alibaba) | $746 ($207–$1,440) | 10% (5–22) | $4,333 | $9,379 | 48 | | gpt-oss-120b (OpenAI · free to download) | $1,929 ($1,346–$2,611) | 40% (32–48) | $1,250 | $27,450 | 131 | Not every mistake is equal. The pre-2022 RMD table mistake is rare (11 times) but large: a typical miss of $5,675 a year. “right IRMAA tier, wrong cents” happened 12 times and is worth about $1.20. Typical miss is the median size of a wrong answer; ranges are 95% intervals. #### Right answers per dollar How often each model was right, against what each right answer cost me. Higher and further left is better. A cheap model isn’t always cheap per answer: some think so long they run out of room and never answer. **Chart: Accuracy against cost per right answer** | Item | Value | |---|---| | Claude Sonnet 5.5 | 84% (77%–89%), $13.27 per 1,000 right answers | | Claude Haiku 5.5 | 77% (70%–83%), $1.29 per 1,000 right answers | | DeepSeek V4.1 Flash | 75% (68%–80%), $5.51 per 1,000 right answers | | Claude Haiku 5.5 (second way) | 73% (66%–78%), $1.34 per 1,000 right answers | | GLM 5.3 Flash | 62% (55%–68%), $3.74 per 1,000 right answers | | Qwen3.8 Flash | 45% (37%–52%), $20.23 per 1,000 right answers | | gpt-oss-120b | 39% (32%–45%), $6.06 per 1,000 right answers | Closed weights Open weights (free to download) #### The four kinds of question Every question is generated from a household’s numbers, so none can be memorized from the web. Open a card to see a real one. The score on each card is Claude Sonnet 5.5’s. One rule 97% One rule at a time: a tax bracket, an RMD divisor, a Social Security claiming factor, a Medicare premium tier. `bracket_room` `federal_tax` `irmaa` `rmd` `ss_claim` `taxable_ss` **A full 2026 federal return, in one step** In 2026, a married couple filing jointly (both spouses under age 65) has $180,000 of wages, $90,000 of traditional IRA distributions, $0 of long-term capital gains and qualified dividends, and $0 of Social Security benefits. They take the standard deduction and have no credits. What is their total federal income tax including any Net Investment Income Tax, rounded to the nearest dollar? - Answer key **$42,268** - Haiku 5.5 **$42,268 ✓** - Sonnet 5.5 **$42,268 ✓** - Haiku 5.5 (second way) **$42,268 ✓** - DeepSeek V4.1 Flash **Ran out of room ✗** - gpt-oss-120b **$42,766 ✗** - Qwen3.8 Flash **Ran out of room ✗** - GLM 5.3 Flash **$42,268 ✓** A few steps 90% Two rules that interact, like a Roth conversion that also makes more Social Security taxable (the “tax torpedo”). `tax_torpedo` `widow_penalty` **A Roth conversion that also makes Social Security taxable** In 2026, a single filer (age 65 or older) has $37,000 of traditional IRA distributions, $21,000 of long-term capital gains and qualified dividends, and $28,100 of Social Security benefits, with no wages. They take the standard deduction and have no credits. If they also convert $36,000 from a traditional IRA to a Roth IRA in 2026, by how many dollars does their total federal income tax (including any Net Investment Income Tax) go up? Round to the nearest dollar. - Answer key **$8,915** - Haiku 5.5 **$8,915 ✓** - Sonnet 5.5 **$8,915 ✓** - Haiku 5.5 (second way) **$8,915 ✓** - DeepSeek V4.1 Flash **$8,915 ✓** - gpt-oss-120b **Ran out of room ✗** - Qwen3.8 Flash **Ran out of room ✗** - GLM 5.3 Flash **$8,915 ✓** Several rules at once 37% Tax, Social Security, the senior deduction and Medicare’s income-based premiums (IRMAA) meeting in one household. `conversion_all_in` `senior_phaseout` `widow_all_in` **Retirees inside the new senior-deduction phase-out** In 2026, a married couple filing jointly (both spouses age 65 or older) has $105,500 of traditional IRA distributions, $0 of long-term capital gains and qualified dividends, and $72,300 of Social Security benefits, with no wages. They take the standard deduction and have no credits. What is their total federal income tax including any Net Investment Income Tax, rounded to the nearest dollar? - Answer key **$16,152** - Haiku 5.5 **$15,928 ✗** - Sonnet 5.5 **$15,928 ✗** - Haiku 5.5 (second way) **$15,928 ✗** - DeepSeek V4.1 Flash **$15,928 ✗** - gpt-oss-120b **$13,511 ✗** - Qwen3.8 Flash **No answer** - GLM 5.3 Flash **$15,928 ✗** Compliance review 100% A planted problem a reviewer must catch: a rollover issue, a missed sales-charge discount, a misleading sentence. `breakpoint` `comms_review` `rollover_flags` **Spotting misleading sentences in an email to clients** [The securities industry’s rule on advertising to clients] says retail communications may not contain any false, exaggerated, unwarranted, promissory or misleading statement or claim, and may not predict or project performance. A representative drafted this email to retail clients about a stock mutual fund: 1. Our analysts project the fund will return 15% next year. 2. CDs are FDIC-insured up to $250,000 per depositor; mutual fund shares are not FDIC-insured. 3. We can meet to review whether this fund fits your goals and time horizon. 4. The fund's expense ratio is 0.65% a year, which is taken out of fund assets. 5. You will earn at least 10% a year in this fund. 6. Past performance does not guarantee future results. Which numbered sentences violate [that rule]? Rule numbers are replaced with plain words in brackets; the exact wording is in data/cases.jsonl in the code. - Answer key **Sentences 1, 5** - Haiku 5.5 **Sentences 1, 5 ✓** - Sonnet 5.5 **Sentences 1, 5 ✓** - Haiku 5.5 (second way) **Sentences 1, 5 ✓** - DeepSeek V4.1 Flash **Sentences 1, 5 ✓** - gpt-oss-120b **Sentences 1, 5 ✓** - Qwen3.8 Flash **Sentences 1, 5 ✓** - GLM 5.3 Flash **Sentences 1, 5 ✓** **How the first test was run** Claude models were asked 10 questions per task through Anthropic’s API, with up to 16,000 tokens per reply; the others 15 per task through OpenRouter (a service that relays requests to many models), with up to 32,000. Reasoning effort was high for all. Each outside model was pinned to one named host with data collection off; “fp8” and “bf16” say how compressed that host’s copy is. Route and host for each row (Claude Haiku 5.5’s “second way” is the OpenRouter route): - **Claude Haiku 5.5** Anthropic API - **Claude Sonnet 5.5** Anthropic API - **Claude Haiku 5.5 (second way)** OpenRouter → Anthropic - **DeepSeek V4.1 Flash** OpenRouter → DeepInfra (fp8) - **gpt-oss-120b** OpenRouter → DeepInfra (bf16) - **Qwen3.8 Flash** OpenRouter → Alibaba - **GLM 5.3 Flash** OpenRouter → Z.ai (fp8) Ranges are 95% intervals; when two overlap heavily, the gap between those models may be noise. Newer, harder set · November 2026 ### The newer, harder set: it depends which AI you ask A month later I wrote 42 harder questions, each making two or more rules meet: the largest Roth conversion (paying tax now to move savings into an account that’s tax-free later) that won’t raise Medicare premiums two years later, a widow’s first year of taxes, Arizona state tax for retirees. I tested 11 models, including OpenAI’s GPT and Google’s Gemini. **Questions per model:** all 42, most models run twice and averaged · **Models:** 11 from seven companies, in three price groups (top-priced, mid-priced, cheap and free); the grey line under each name says which · **Settings:** low reasoning effort, replies up to 4,000 tokens · **Graded:** by a script against the answer key, which a second, separately written engine must match **Different settings from the first test.** Every model ran at low reasoning effort with a 4,000-token reply limit, a quick answer rather than a long think. Don’t compare these scores with the October ones. | Model | Right, of the newer set | |---|---| | GPT-6.1 Sol (OpenAI · top-priced) | 98% · 41/42 (88–100) | | GPT-6.1 Terra (OpenAI · mid-priced) | 80% · 33.5/42 (65–89) | | Claude Sonnet 5.5 (Anthropic · top-priced) | 62% · 26/42 (47–75) | | DeepSeek V4.1 Flash (DeepSeek · cheap and free) | 29% · 12/42 (17–44) | | Llama 4 Maverick (Meta · cheap and free) | 6% · 2.5/42 (2–17) | The spread is the finding: GPT-6.1 Sol got 98%, GPT-6.1 Terra 80%, Claude Sonnet 5.5 62%, and all 5 cheap and free models under 30%. The small grey numbers and the shaded band behind each bar show the range the true score likely falls in, given only 42 questions; most models ran twice, so a half (like 33.5) means the two runs differed. A low score can mean wrong math or simply running out of room: DeepSeek V4.1 Flash hit the reply limit before answering on 71% of questions, which counts as not right. **All models and columns (for the detail-minded)** | Model | Right | Ran out of room | Runs averaged | Cost per 1,000 right | |---|---|---|---|---| | GPT-6.1 Sol (OpenAI · top-priced) | 98% · 41/42 (88–100) | – | 1 | $3.22 | | GPT-6.1 Terra (OpenAI · mid-priced) | 80% · 33.5/42 (65–89) | – | 2 | $1.89 | | Claude Sonnet 5.5 (Anthropic · top-priced) | 62% · 26/42 (47–75) | – | 1 | $17.33 | | Claude Haiku 5.5 (Anthropic · mid-priced) | 52% · 22/42 (38–67) | 2% | 2 | $2.21 | | Gemini 3.1 Pro (Google · top-priced) | 29% · 12/42 (17–44) | – | 1 | $51.35 | | Gemini 3.8 Flash (Google · mid-priced) | 29% · 12/42 (17–44) | 36% | 1 | $19.47 | | DeepSeek V4.1 Flash (DeepSeek · cheap and free) | 29% · 12/42 (17–44) | 71% | 2 | $3.64 | | GLM 5.3 Flash (Z.ai · cheap and free) | 15% · 6.5/42 (8–29) | 20% | 2 | $6.30 | | gpt-oss-120b (OpenAI · cheap and free) | 7% · 3/42 (2–19) | – | 2 | $1.60 | | Mistral Small (2603) (Mistral · cheap and free) | 7% · 3/42 (2–19) | – | 2 | $8.73 | | Llama 4 Maverick (Meta · cheap and free) | 6% · 2.5/42 (2–17) | – | 2 | $18.47 | Most models were run twice and the two runs averaged, so a half point in the Right column means the two runs differed. Ranges are 95% intervals; with 42 questions they are wide, so the order of models close together is not reliable. How the answers were checked is on [Technical details](https://plannerbench.org/method.html). GPT-6 Luna through Codex · November 2026 ### GPT-6 Luna through Codex (the model on my ChatGPT subscription, no code or web): often right on one rule, almost never when rules meet Most people don’t reach an AI through a developer account; they type into ChatGPT, whose regular app uses GPT-6 Luna, a different, cheaper model than GPT-6.1 Sol. So I asked GPT-6 Luna the same questions through OpenAI’s Codex app on my own subscription, and graded its replies with the same answer key and the same grading script as everyone else. That’s close to, but not the same as, typing into the ChatGPT window, so these results describe this setup. **Questions:** 139 of the 140 first-test questions the Claude models got (one went unanswered), the 42 of the newer set, 30 wrong-facts files and 11 whole-household decisions · **Settings:** medium reasoning effort, no reply limit, no code, calculator or web · **Graded:** by the same script against the same answer key; wrong facts by the same AI judge | Questions | GPT-6 Luna | |---|---| | First test · one rule | 71% · 42 of 59 | | First test · the 50 that chain or stack rules | 2% · 1 of 50 | | Newer, harder set | 17% · 7 of 42 | | Wrong facts · caught the planted mistake | 50% · 15 of 30 | | Whole-household decisions | declined all 10 answerable ones | On one-rule questions it was right about 71% of the time. Once rules had to work together, it almost never was. On the whole-household decisions it didn’t try: it answered that each one needed a full multi-year projection, and it did correctly turn down the one question that can’t be answered. On wrong facts it caught half the planted mistakes and gave a number anyway on 40% without saying anything looked wrong. **Why it isn’t in the tables above.** The setup differs. The other models got one message per question, at low reasoning effort with a 4,000-token reply limit. GPT-6 Luna ran at medium effort with no limit, and I sent the questions in small batches, each to be answered as its own chat. In the regular ChatGPT app the model can choose to run code; here it was told not to, like every other model on this page. **How the GPT-6 Luna run was done** The questions went through Codex, OpenAI’s coding app, which runs on a ChatGPT subscription, so no per-question fee and no developer account were involved. Each batch had a handful of questions with their answers removed and their names replaced by plain codes, and the instruction to answer each as its own chat, showing the reasoning, with no code, calculator, web search or other files. A first attempt was thrown out because most replies were a bare final number with no reasoning, which isn’t how ChatGPT answers a person. In the second attempt, one session refused a whole batch and some questions were skipped; those were asked again in a fill-in batch, and one first-test question still went unanswered, so its first-test score is out of 139. Within a batch, earlier questions stayed in the conversation, which a fresh chat wouldn’t have. Ranges (95%): newer set 8–31%; one rule 59–81%. Grading checks are on [Technical details](https://plannerbench.org/method.html#ai-work). ### What these results don’t say - **Most wrong answers are unexplained.** Beyond the senior-deduction misreading, most matched no known mistake, so I can’t say what caused them. - **It isn’t one company’s blind spot.** In the first test, DeepSeek and GLM made the same senior-deduction misreading as Claude. Some misses show models using an older version of a rule: gpt-oss-120b used the withdrawal table replaced in 2022 on every withdrawal question it got wrong. - **It’s still small.** 10–15 questions per task per model in the first test, 42 in the newer set. A gap of a few points is within the noise; the ranges show how much. [Next **With a calculator: what happens when AI can call one it trusts**](https://plannerbench.org/calculator.html) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## Give it a calculator it can trust, and some models get it right. Not all. Source: https://plannerbench.org/calculator.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt Finding 2 · With a calculator The plain answer I turned my answer key into a calculator an AI model can use while it answers. It helps a lot, but only if the calculator shows its work and the model actually uses it well. It rescued some models, including a cheap one, and did nothing for others. One example: it argued with the calculator Claude Haiku 5.5 asked the calculator for a couple’s tax, got the right number, and replied: > “The calculation tool's figure of $21,639 applies the senior-deduction phase-out to each spouse separately, which I don't think the statute supports.” It used its own wrong reading of a new 2025 tax break for people over 65 instead, and got the tax wrong. When the calculator also showed the rule and the IRS form behind the number, it stopped arguing. **What I mean by a “calculator”** In AI, a “tool” is a program a model can call in the middle of an answer, the way you’d reach for a calculator. Mine is the planning engine: code for 2026 federal tax, Social Security, required withdrawals from retirement accounts and Medicare premiums, checked against IRS, Social Security and Medicare publications. The model still has to decide what to type in and how to put the pieces together, which is where it can still go wrong. Three calculator tests on this page 1. [**Calculator test 1: number vs rule**](https://plannerbench.org/calculator.html#tools) (October). One model, the hardest first-test questions. A right number wasn’t enough; the rule behind it was. 2. [**Calculator test 2: who it rescues**](https://plannerbench.org/calculator.html#cheap) (November). Eight models on the newer, harder set. A cheap model jumped; three others didn’t move. 3. [**Calculator test 3: whole-household decisions**](https://plannerbench.org/calculator.html#gauntlet). Real choices over 25 years, three models. Plus GPT-6 Luna through Codex (the model on my ChatGPT subscription) without a calculator, and a short test: leave a fact out, and see if it asks. Calculator test 1 of 3 · October 2026 ### Hand it the right number, and it argues. Show it the rule, and it listens. - **Questions**: the 50 hardest from the first test, plus 50 held-back ones - **Model**: Claude Haiku 5.5 - **Asked**: three ways: on its own; with calculator 1.0 (sends back only the number); with 1.1 (the number, plus the rule and the IRS form line behind it) - **Settings**: high reasoning effort (lets the model think longer), long replies, one run per question - **Graded**: by code, against the answer key, within $1–2 - **Held out**: yes: the held-back questions were kept secret and never used while building the calculator **Result.** 40% on its own, 62% with only the number, 100% with the rule shown. On the held-back questions: 44% → 96%. The calculator’s numbers were the same in both versions; only what it said about them changed. | Claude Haiku 5.5 | The hardest questions | Held-back questions | Dollars off per question, per year | |---|---|---|---| | On its own | 40% · 20/50 (28–54) | 44% · 22/50 (31–58) | $318 (held-back $216) | | Calculator 1.0: just the number | 62% · 31/50 (48–74) | (not asked) | $229 | | Calculator 1.1: the number, the rule and its source | 100% · 50/50 (93–100) | 96% · 48/50 (87–99) | $0 (held-back $4) | The same 50 questions in every row (5 tasks, 10 each). The right number alone fixed 11 questions it had wrong and broke 0 it had right; adding the rule and its source fixed 19 more and broke 0. With the same questions asked both ways, a jump that size is very unlikely to be luck (the statistics are under “For the technical reader” below). On 50 held-back questions, never seen while the calculator was built: 26 fixed, 0 broken. The shaded band behind each bar is the range the true score likely falls in, given so few questions. The question from the example at the top - **The calculator’s answer (and the key)**: $21,639 - **Haiku with calculator 1.0**: $21,121 (Wrong) - **Haiku with calculator 1.1**: $21,639 (Right) What version 1.1 sent back with the tax 1. Income (MAGI) over the $150,000 threshold — $39,225.00 2. Reduction per person (6%) — $2,353.50 3. Senior deduction per person — $3,646.50 4. People 65 or older — 2 5. Senior deduction — $7,293.00 The senior deduction from the 2025 tax law (OBBBA), IRS Form 1040 Schedule 1-A Part V (lines 35-36b): $6,000 for each person 65 or older, reduced by 6% of MAGI over $150,000. The reduction is figured for each qualifying person separately: on a joint return each spouse's $6,000 is reduced by the full 6% of the excess, not one reduction off a combined $12,000. - **A right number wasn’t enough.** Across all 140 questions, calculator 1.0 lifted Haiku from 77% to 84% (13 fixed, 3 broken, p = 0.021); single-step questions were already at 97%. But 18 of the 22 misses left were Haiku getting the right number from the calculator and replacing it with its own, every time saying so. - **It held on questions nobody tuned against.** On 50 questions held back from a secret seed and never used while building the calculator: 44% on its own, 96% with calculator 1.1. Both misses were Haiku overruling the calculator again, with the rule in front of it. - **Limits.** One model, one run per question. The held-back questions came from the same generator, so they test the same kinds of rules. These are the hardest questions, which is why Haiku scores lower here than it does overall. Calculator test 2 of 3 · November 2026 ### Who it rescues: a cheap model jumped, three others didn’t move - **Questions**: the 42 questions in the newer, harder set, each with several rules meeting in one household - **Models**: 8 of the 11 models, on their own and with the calculator (Claude Sonnet 5.5 and the two Gemini models weren’t run with it, to save money) - **Settings**: low reasoning effort; replies capped at 4,000 tokens (a token is about three-quarters of a word); with the calculator, the cap applied to each of up to 8 back-and-forth turns - **Compared**: one run with the calculator against the first run on its own, question by question - **Graded**: by code, against the answer key, within $2–5 - **Held out**: no: the newer set has no secret questions **Result.** DeepSeek V4.1 Flash, a cheap model, went from 29% to 79% with the calculator setup: 21 questions fixed, none broken. That’s higher than Claude Sonnet 5.5 on its own (62%), at about 1/15 of the cost per right answer ($1.18 per 1,000 right answers against $17.33). **Not all of that gain is the calculator.** The setup also let a model write more in total, because the reply cap applied to each turn. On its own, DeepSeek hit the cap before answering on 71% of questions; with the calculator, 9 of its replies went past 4,000 tokens in total. So part of the jump may come from that extra room. The conservative number: counting only replies that stayed within 4,000 tokens in total, it got 27/42 (64%). Higher than Sonnet’s score, but on only 42 questions that lead could still be luck. | Model | On its own (first run) | With the calculator (one run) | |---|---|---| | GPT-6.1 Sol (top-priced · one run each way) | 98% · 41/42 (88–100) | 95% · 40/42 (84–99) | | DeepSeek V4.1 Flash (cheap and free · one run each way) | 29% · 12/42 (17–44) | 79% · 33/42 (64–88) | | Claude Haiku 5.5 (mid-priced · one run each way) | 48% · 20/42 (33–62) | 62% · 26/42 (47–75) | | gpt-oss-120b (cheap and free · one run each way) | 7% · 3/42 (2–19) | 10% · 4/42 (4–22) | | Claude Sonnet 5.5 (top-priced · reference: not run with the calculator) | 62% · 26/42 (47–75) | (not run) | Four of the 8 models run both ways, plus Claude Sonnet 5.5 on its own for reference; all of them are in the first table under “All models and columns” below. Both columns are one run per question, on the same 42 questions. Highlighted: the calculator added more than 30 points. The small grey numbers and the shaded band behind each bar show the range the true score likely falls in, given so few questions. These settings differ from calculator test 1 above, so don’t compare the two sets of scores. Called it, then wrong A single retiree’s tax, with a pension, Social Security, and a required withdrawal partly given to charity. gpt-oss-120b made one call with the pension alone, left out the withdrawal, and replied with one line: > “ANSWER: 1115” The right answer is $9,614. Right parts, wrong assembly GPT-6.1 Sol took the federal numbers from the calculator for an Arizona tax question, then did the state part itself: > “Arizona’s standard deduction does not include the federal age-65 addition… $17,900 × 2.5% = $447.50. ANSWER: 448” It missed Arizona’s separate $2,100 exemption for people 65 and older. The right answer is $395. The calculator gave it correct parts; putting them together was its own work. - **Three cheap models called it and stayed low.** gpt-oss-120b, Llama 4 Maverick and Mistral Small called the calculator on 79% of the questions and still got only 2–4 of 42 right: wrong numbers in, or one call where the question needed three. - **The highest scorer gained nothing.** GPT-6.1 Sol got 41/42 on its own and 40/42 with it. - **Claude Haiku 5.5 skipped it half the time.** It called the calculator on 50% of questions, and once said so outright: “I did not run this through the calculator tools.” Its gain is small enough to be luck. **All models and columns (for the detail-minded)** Every model run both ways, one run each. Fixed: wrong on its own, right with the calculator; broken: the reverse. Called it: the share of questions where the model used the calculator at all. | Model | On its own (first run) | With the calculator (one run) | Fixed / broken | Called it | |---|---|---|---|---| | GPT-6.1 Sol (top-priced · one run each way) | 98% · 41/42 (88–100) | 95% · 40/42 (84–99) | 0 / 1 | 79% | | GPT-6.1 Terra (mid-priced · one run each way) | 81% · 34/42 (67–90) | 86% · 36/42 (72–93) | 3 / 1 | 81% | | DeepSeek V4.1 Flash (cheap and free · one run each way) | 29% · 12/42 (17–44) | 79% · 33/42 (64–88) | 21 / 0 | 90% | | Claude Haiku 5.5 (mid-priced · one run each way) | 48% · 20/42 (33–62) | 62% · 26/42 (47–75) | 8 / 2 | 50% | | GLM 5.3 Flash (cheap and free · one run each way) | 17% · 7/42 (8–31) | 43% · 18/42 (29–58) | 11 / 0 | 86% | | gpt-oss-120b (cheap and free · one run each way) | 7% · 3/42 (2–19) | 10% · 4/42 (4–22) | 2 / 1 | 79% | | Mistral Small (2603) (cheap and free · one run each way) | 7% · 3/42 (2–19) | 7% · 3/42 (2–19) | 3 / 3 | 79% | | Llama 4 Maverick (cheap and free · one run each way) | 7% · 3/42 (2–19) | 5% · 2/42 (1–16) | 0 / 1 | 79% | | Claude Sonnet 5.5 (top-priced · reference: not run with the calculator) | 62% · 26/42 (47–75) | (not run) | – | – | Every model in the November set, with every run averaged: | Model | On its own | With the calculator | Called it | Cost per 1,000 right, on its own → with it | |---|---|---|---|---| | GPT-6.1 Sol (top-priced · one run each way) | 98% (88–100) | 95% (84–99) | 79% | $3.22 → $3.19 | | GPT-6.1 Terra (mid-priced · on its own: 2 runs averaged · with it: 1 run) | 80% (65–89) | 86% (72–93) | 81% | $1.89 → $1.79 | | DeepSeek V4.1 Flash (cheap and free · on its own: 2 runs averaged · with it: 1 run) | 29% (17–44) | 79% (64–88) | 90% | $3.64 → $1.18 | | Claude Sonnet 5.5 (top-priced · 1 run per item) | 62% (47–75) | (not run) | – | $17.33 | | Claude Haiku 5.5 (mid-priced · on its own: 2 runs averaged · with it: 1 run) | 52% (38–67) | 62% (47–75) | 50% | $2.21 → $3.63 | | GLM 5.3 Flash (cheap and free · on its own: 2 runs averaged · with it: 1 run) | 15% (8–29) | 43% (29–58) | 86% | $6.30 → $2.46 | | Gemini 3.1 Pro (top-priced · 1 run per item) | 29% (17–44) | (not run) | – | $51.35 | | Gemini 3.8 Flash (mid-priced · 1 run per item) | 29% (17–44) | (not run) | – | $19.47 | | gpt-oss-120b (cheap and free · on its own: 2 runs averaged · with it: 1 run) | 7% (2–19) | 10% (4–22) | 79% | $1.60 → $5.02 | | Mistral Small (2603) (cheap and free · on its own: 2 runs averaged · with it: 1 run) | 7% (2–19) | 7% (2–19) | 79% | $8.73 → $11.75 | | Llama 4 Maverick (cheap and free · on its own: 2 runs averaged · with it: 1 run) | 6% (2–17) | 5% (1–16) | 79% | $18.47 → $62.65 | Unlike the table above, “On its own” here averages every run of a question (two for most models, one for the most expensive), so Claude Haiku 5.5 shows 52% here and 20/42 (its first run) above. Each row says how many runs went into each column. Price group is by the model’s price per token: top-priced, mid-priced, or cheap and free. **By kind of question (for the detail-minded)** | Model | Roth conversion, Medicare-safe | Roth conversion cost with Arizona tax | Taxable Social Security | Inherited IRA withdrawal | Charity from a required withdrawal | Widow’s first year | Arizona tax | |---|---|---|---|---|---|---|---| | GPT-6.1 Sol (top-priced · on its own → with it) | 100% → 100% | 100% → 100% | 100% → 100% | 100% → 100% | 100% → 100% | 100% → 100% | 83% → 67% | | GPT-6.1 Terra (mid-priced · on its own → with it) | 100% → 100% | 75% → 100% | 100% → 100% | 92% → 83% | 92% → 100% | 100% → 100% | 0% → 17% | | DeepSeek V4.1 Flash (cheap and free · on its own → with it) | 67% → 100% | 0% → 83% | 100% → 100% | 33% → 50% | 0% → 100% | 0% → 83% | 0% → 33% | | Claude Sonnet 5.5 (top-priced · on its own) | 100% | 17% | 100% | 83% | 67% | 67% | 0% | | Claude Haiku 5.5 (mid-priced · on its own → with it) | 100% → 100% | 8% → 50% | 92% → 83% | 33% → 50% | 67% → 83% | 67% → 67% | 0% → 0% | | GLM 5.3 Flash (cheap and free · on its own → with it) | 42% → 100% | 0% → 0% | 33% → 50% | 25% → 33% | 0% → 67% | 8% → 50% | 0% → 0% | | Gemini 3.1 Pro (top-priced · on its own) | 17% | 0% | 100% | 83% | 0% | 0% | 0% | | Gemini 3.8 Flash (mid-priced · on its own) | 0% | 0% | 100% | 67% | 33% | 0% | 0% | | gpt-oss-120b (cheap and free · on its own → with it) | 0% → 0% | 0% → 17% | 17% → 0% | 33% → 33% | 0% → 17% | 0% → 0% | 0% → 0% | | Mistral Small (2603) (cheap and free · on its own → with it) | 0% → 0% | 0% → 0% | 17% → 50% | 25% → 0% | 8% → 0% | 0% → 0% | 0% → 0% | | Llama 4 Maverick (cheap and free · on its own → with it) | 0% → 0% | 0% → 0% | 8% → 0% | 33% → 33% | 0% → 0% | 0% → 0% | 0% → 0% | Six questions of each kind. Each cell is right on its own (every run averaged) → right with the calculator. The kinds: the largest Roth conversion (moving money from a regular IRA to a Roth IRA and paying the tax now) that keeps Medicare premiums level; what a conversion costs in federal plus Arizona tax; how much Social Security is taxed when there’s tax-free bond interest and a gift to charity from an IRA; this year’s required withdrawal from an inherited IRA; tax when part of a required withdrawal goes to charity; a widow’s first tax year; Arizona tax for retirees. Calculator test 3 of 3 · Whole-household decisions ### Real retirement decisions: with the calculator, all three models made the right decision, with every number within 2%, every time - **Questions**: 11 whole made-up households (about 16 facts each: Social Security, IRAs, a pension, savings), one from each of 10 kinds of decision plus a trap with no honest answer - **Models**: Claude Sonnet 5.5, Claude Haiku 5.5, DeepSeek V4.1 Flash - **Settings**: high reasoning effort; reply budget 64,000 tokens for the Claude models and 32,000 for DeepSeek; one try each - **Graded**: by code: “fully right” means the right decision and every number within 2% of the key - **Held out**: no: 33 held-back questions are built but not yet run **Result.** Each choice is one people really face: how much to convert to a Roth, when to claim Social Security, how to give to charity, how fast to empty an IRA inherited from someone who died. Getting it right means combining at least three rules over 25 years. With the calculator, all 3 models got every question fully right: the right decision, and every number within 2% of the key. On its own, DeepSeek V4.1 Flash got 2/11. | Model | Fully right | Ran out of room | Thinking per question | Cost per question | |---|---|---|---|---| | Claude Haiku 5.5, on its own | 36% · 4/11 (15–65) | 0 | 34k tokens | $0.02 | | Claude Sonnet 5.5, on its own | 70% · 7/10 (40–89) | 3 | 37k tokens | $0.38 | | DeepSeek V4.1 Flash, on its own | 18% · 2/11 (5–48) | 0 | 27k tokens | $0.01 | | Claude Haiku 5.5 + calculator | 100% · 11/11 (74–100) | 0 | 9k tokens | $0.01 | | Claude Sonnet 5.5 + calculator | 100% · 11/11 (74–100) | 0 | 3k tokens | $0.16 | | DeepSeek V4.1 Flash + calculator | 100% · 11/11 (74–100) | 0 | 10k tokens | $0.01 | 11 questions, one from each family plus a trap whose honest answer is “I can’t answer that.” Ran out of room: used its whole reply budget thinking and never answered (scored as not right). Sonnet alone shows 10 because one reply was cancelled to stay inside the run’s spending cap. The shaded band behind each bar is the range the true score likely falls in, given so few questions. One question, both ways Ray and Linda want to start Roth conversions from Ray's traditional IRA this year and want to know how much to convert. - **The calculator’s best plan**: $118,000 a year for 8 years (2026–2033) - **DeepSeek V4.1 Flash, on its own**: Convert nothing (a projected $134,445 less after-tax wealth over 25 years; a simulation, not a real household) (Wrong) - **DeepSeek V4.1 Flash + calculator**: $118,000 a year × 8 years (Right) Confident, wrong, and the projected after-tax wealth it gives up over 25 years (simulated households, not real ones) - 1: How much to convert to Roth, and for how many years (DeepSeek V4.1 Flash, on its own). With the calculator: right. — −$134,445 - 2: Converting inside the senior-deduction phase-out (DeepSeek V4.1 Flash, on its own). With the calculator: right. — −$72,529 - 3: How much to convert to Roth, and for how many years (Claude Haiku 5.5, on its own). With the calculator: right. — −$26,002 - **The calculator’s answers held.** With the calculator, all 3 models made the right decision on every question, with every number within 2% of the key, and never went against it. They also thought far less: about 3k tokens for Sonnet instead of 37k. - **Hard on purpose.** Code checks that no single tax rate or skipped rule lands within 2% of the answer. The answers are rechecked by separate programs (203 of 203 agree; [Technical details](https://plannerbench.org/method.html)). - **Still small.** One try per question, three models. The intervals are wide on purpose: this shows the gap, not a ranking. A separate result, not in the table above #### GPT-6 Luna through Codex, without a calculator: it said it couldn’t - **Questions**: the same 11 households, trap included - **Model**: GPT-6 Luna through Codex (the model on my ChatGPT subscription, run in OpenAI’s Codex app), with no calculator, code or web - **Settings**: medium effort, no length limit, one try each; a different setup from the three models above - **Graded**: the same way: the right decision, and every number within 2% **Result.** It declined all 10 questions that had an answer, typically like this: “A reliable answer requires a full multi-year projection of taxes, RMDs, premiums, and account balances.” (RMDs are the withdrawals the IRS requires from retirement accounts.) It also rightly declined the trap, so it scored 1 of 11. **Why this matters.** That’s an honest answer, and it’s the case for a calculator. A real decision needs a 25-year projection. GPT-6 Luna declined every one; some other models answered some of these correctly without a calculator (Claude Sonnet 5.5 got 7 of 10), and with one, all three models tested got every decision right. #### Short test: leave a fact out. Does it ask, or guess? - **Questions**: the same household questions (not the trap), each with the one deciding fact left out: a Social Security amount or an IRA balance. The file says something may be missing, not what. - **Model**: DeepSeek V4.1 Flash, on its own and with the calculator - **Settings**: as above; the model may ask a scripted client up to three questions before answering - **Graded**: the same way, against the full facts **Result.** With the calculator, DeepSeek asked for the missing fact 9 times out of 10. | Model | Fully right | Asked for the missing fact | Cost per question | |---|---|---|---| | DeepSeek V4.1 Flash, on its own | 40% · 2/5 (12–77) | 5 of 5 | $0.01 | | DeepSeek V4.1 Flash + calculator | 90% · 9/10 (60–98) | 9 of 10 | $0.01 | One question per family (the trap has nothing to leave out). The alone run was stopped after 5 questions: answering without the calculator took 10–15 minutes each. Fully right is graded against the full facts, by the same grader as above. The shaded band behind each bar is the range the true score likely falls in, given so few questions. What it asked (word for word) - 1: “Is Frank receiving Social Security now—and if so, what is his monthly benefit and at what age did he claim? If he is not yet claiming, what is his monthly benefit at full retirement age and when does he plan to claim?” - 2: “The options call for converting from Glen's traditional IRA, but the account list you gave me only has Maria's traditional IRA ($70,000) and Glen's Roth IRA — there's no Glen traditional IRA or 401(k) listed. …” - 3: “What is Ken's traditional IRA balance as of December 31, 2025, and is any of it after-tax money (i.e., does he have a Form 8606 basis)?” Why guessing can’t pass - **Every tested guess fails.** Before a question is used, the engine answers it 4–5 times with the missing fact guessed (none, half, 75%, 125%, 150%). All 69 of 69 questions grade wrong under every guess. - **The client never volunteers.** It’s code, not a model: it answers only a question on the missing fact’s topic, and says nothing new to anything else. - **Checked with stand-ins.** A stand-in that asks the right question gets 69 of 69 fully right; one that guesses gets 0 of 69; one that asks vague questions never gets the fact (0 of 69). See it work ### Watch the calls, or see it in a chat - [**Replays**](https://plannerbench.org/replay.html): each household decision above, step by step: the household, every calculator call, and the answer against the key. - [**Demo walkthrough**](https://plannerbench.org/chat.html): the same calculator as a private add-on for Claude and ChatGPT. Not open to the public; that page shows its real replies. **For the technical reader** The calculator is an MCP server (the Model Context Protocol, the standard way to plug tools into AI models). The benchmark runners and the chat add-on call the same code. The first test offered federal tax, taxable Social Security, required withdrawals, Social Security benefits, Medicare premiums and tax tables. The November set offered those plus a year-by-year scenario runner; the Roth-conversion solver and the chat front door were left out to keep the instructions short. **Calculator test 1**: from calculator 1.0 (just the number) to 1.1 (the number plus the rule), the same questions both ways; exact McNemar test, p < 0.001. Shaded bands in every table are 95% intervals (Wilson). **Calculator test 2** (the 42 questions in the newer set; low effort; one run each way; graded by code within $2–5). Token budget: 4,000 output tokens per turn, up to 8 turns. 10 of 336 calculator replies used more than 4,000 in total, the longest 7,135. DeepSeek V4.1 Flash with the calculator against Claude Sonnet 5.5 on its own, the same 42 questions, first run each: 10 only DeepSeek got right, 3 only Sonnet got right; exact McNemar p ≈ 0.0923. Higher observed score, not a significant difference. DeepSeek alone → with: 21 fixed, 0 broken, p < 0.001. Claude Haiku 5.5: 8 fixed, 2 broken, p = 0.1094. First test: Claude Haiku 5.5 through Anthropic’s API. November set: every model through OpenRouter (a service that relays requests to many models), each pinned to one host. The calculator prices each choice; it never picks investments or products. [Next **Wrong facts: what happens when the file you give it has a mistake in it**](https://plannerbench.org/wrong-facts.html) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## Tell it something wrong: GPT-6.1 Sol and Claude Sonnet 5.5 caught most mistakes. Three cheap models caught almost none. Source: https://plannerbench.org/wrong-facts.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt Finding 3 · Wrong facts · the newer, harder set (November 2026) The plain answer I gave 11 AI models client files with one fact that couldn’t be right, and measured two things. **Recognized the mistake:** did the reply name it? **Stopped and asked:** did it ask for the right fact instead of giving a number? It counts as **caught** only when it did both. The two best at catching mistakes without false alarms, GPT-6.1 Sol and Claude Sonnet 5.5, caught 80% and 73% while almost never refusing a file that was fine. Three of the five cheap and free models (gpt-oss-120b, Llama 4 Maverick, Mistral Small) caught 1–3% and handed back a confident number 81–93% of the time. **GPT-6 Luna through Codex** (the model on my ChatGPT subscription), tested separately, caught 50%. | Of 30 planted mistakes | Recognized it | Stopped and asked | Caught (both) | Refused a fine file | |---|---|---|---|---| | GPT-6.1 Sol (top-priced) | 100% | 80% | **80%** | 3% | | Claude Sonnet 5.5 (top-priced) | 93% | 73% | **73%** | 0% | | Gemini 3.8 Flash (mid-priced) | 92% | 100% | **92%** | 50% | | DeepSeek V4.1 Flash (cheap and free) | 76% | 68% | **61%** | 0% | | gpt-oss-120b (cheap and free) | 3% | 7% | **3%** | 5% | Recognizing isn’t stopping: GPT-6.1 Sol recognized 100% of the mistakes but stopped on 80%. Stopping isn’t recognizing either: Gemini 3.8 Flash stopped on 100% of the files with a mistake, but also refused 50% of the fine ones (out of 40), so a refusal from it tells you little. All 11 models are under “All models and columns” below. One example A client file said a retired school administrator with a $785,000 IRA was born in 2012, and asked how much she must withdraw this year. One model (gpt-oss-120b) replied: > “Priya will be 14 years old throughout 2026 … no required minimum distribution is required. ANSWER: 0” It saw that she was 14 and kept going. With her real birth year, the answer is $34,279. Words on this page - **Best at catching mistakes**: On this page, “best” means catching the most mistakes without refusing files that were fine (caught minus false alarms): GPT-6.1 Sol, then Claude Sonnet 5.5. - **Price groups**: By what the model costs per word. **Top-priced:** Claude Sonnet 5.5, GPT-6.1 Sol, Gemini 3.1 Pro. **Mid-priced:** GPT-6.1 Terra, Gemini 3.8 Flash, Claude Haiku 5.5. **Cheap and free:** DeepSeek V4.1 Flash, GLM 5.3 Flash, gpt-oss-120b, Llama 4 Maverick, Mistral Small (2603). - **GPT-6 Luna**: The model the regular ChatGPT app uses, a different, cheaper model than GPT-6.1 Sol. I ran it through OpenAI’s Codex app on my ChatGPT subscription, with code and web off, which is close to but not the same as typing into ChatGPT. It was tested in a different setup, so it gets its own box below and isn’t ranked in the tables. - **Clean twin**: The same file with the mistake taken out, to check the model can answer it when nothing is wrong. - **False alarm**: Refusing to answer a file that was fine. How the test works ### 30 client files with one planted mistake, and 40 that were fine I wrote 30 short client files, the kind a planner jots down at a first meeting, each with one mistake planted in it: two digits swapped in a balance, a birth year that makes a retiree 14, monthly spending ten or twelve times too high (more than the family owns, every year), “married” next to “filing single”, or a contribution over the legal limit. Each one has a **clean twin**: the same file with the mistake fixed, so I can tell a model that spots the error from one that just refuses hard questions. 10 more files look odd but are fine (a 90-year-old IRA owner, a contribution exactly at the limit), so flagging everything unusual doesn’t win. Every file ended the same way: give a number, or say you need more information and what you need. So every model had an easy way out. What the models did ### GPT-6.1 Sol and Claude Sonnet 5.5 caught most mistakes. Three cheap models caught almost none. The two measures split every reply to a file with a planted mistake into four outcomes, the colors in the chart: | | Stopped and asked | Gave a number | |---|---|---| | Recognized the mistake | **Caught it.** What you want. | **Recognized it, answered anyway.** Said something looked wrong, then gave a number built on it. | | Didn’t recognize it | **Declined for another reason**, such as doubting this year’s tax law. | **Missed it, answered anyway.** A confident number. The dangerous one. | A fifth outcome, **ran out of room** (hit the length limit before answering), counts as a failure. **Chart: What each model did with the 30 files that had a mistake planted in them, as shares that add to 100%; the table under the toggle below has the same numbers** ([image](https://plannerbench.org/charts/outcomes.svg)) | Item | Value | |---|---| | Gemini 3.8 Flash | Caught it 92% | | Gemini 3.8 Flash | Declined for another reason 8% | | GPT-6.1 Sol | Caught it 80% | | GPT-6.1 Sol | Recognized it, answered anyway 20% | | Claude Sonnet 5.5 | Caught it 73% | | Claude Sonnet 5.5 | Recognized it, answered anyway 20% | | Claude Sonnet 5.5 | Missed it, answered anyway 7% | | Gemini 3.1 Pro | Caught it 70% | | Gemini 3.1 Pro | Missed it, answered anyway 3% | | Gemini 3.1 Pro | Declined for another reason 27% | | GPT-6.1 Terra | Caught it 67% | | GPT-6.1 Terra | Recognized it, answered anyway 33% | | Claude Haiku 5.5 | Caught it 65% | | Claude Haiku 5.5 | Recognized it, answered anyway 17% | | Claude Haiku 5.5 | Missed it, answered anyway 18% | | DeepSeek V4.1 Flash | Caught it 61% | | DeepSeek V4.1 Flash | Recognized it, answered anyway 14% | | DeepSeek V4.1 Flash | Missed it, answered anyway 18% | | DeepSeek V4.1 Flash | Ran out of room 7% | | GLM 5.3 Flash | Caught it 40% | | GLM 5.3 Flash | Recognized it, answered anyway 31% | | GLM 5.3 Flash | Missed it, answered anyway 14% | | GLM 5.3 Flash | Ran out of room 14% | | Llama 4 Maverick | Caught it 3% | | Llama 4 Maverick | Recognized it, answered anyway 16% | | Llama 4 Maverick | Missed it, answered anyway 81% | | gpt-oss-120b | Caught it 3% | | gpt-oss-120b | Missed it, answered anyway 93% | | gpt-oss-120b | Declined for another reason 3% | | Mistral Small (2603) | Caught it 1% | | Mistral Small (2603) | Recognized it, answered anyway 1% | | Mistral Small (2603) | Missed it, answered anyway 91% | | Mistral Small (2603) | Declined for another reason 7% | **What it shows.** Each bar is one model. Pooling every model, there were 720 replies to a file with a planted mistake: 30 files, asked once of each of the 3 top-priced models, twice of each of the 3 mid-priced models and three times of each of the 5 cheap and free models (cheaper models got more runs within the same budget). Of those replies, 42% caught the mistake and 39% missed it and answered anyway. The split is mostly by model: GPT-6.1 Sol and Claude Sonnet 5.5 stopped most of the time; gpt-oss-120b, Llama 4 Maverick and Mistral Small almost never did. One cheap model, DeepSeek V4.1 Flash, caught 61%, close to the mid-priced models. **Chart: Share of the 40 fine files each model refused to answer; the table under the toggle below has the same numbers** ([image](https://plannerbench.org/charts/false_alarms.svg)) | Item | Value | |---|---| | Mistral Small (2603) | refused 1% of correct notes | | Llama 4 Maverick | refused 2% of correct notes | | GPT-6.1 Sol | refused 3% of correct notes | | GPT-6.1 Terra | refused 3% of correct notes | | gpt-oss-120b | refused 5% of correct notes | | Gemini 3.8 Flash | refused 50% of correct notes | | Gemini 3.1 Pro | refused 68% of correct notes | **What it shows.** The other way to fail is refusing a file that was fine. Most models almost never did. The two Gemini models refused 50% and 68% of the 40 fine files, mostly because they didn’t believe this year’s tax law was real (it was passed after their training ended). So their high catch rates come with a lot of crying wolf, which is why “best” on this page means catching mistakes without refusing fine files. **All models and columns (for the detail-minded)** | Model | Caught it | Recognized it, answered anyway | Missed it, answered anyway | Declined for another reason | Ran out of room | False alarms, of 40 fine files | Clean twins answered right, of 30 | |---|---|---|---|---|---|---|---| | GPT-6.1 Sol (top-priced · 1 run per item) | 80% (63–91) | 20% | 0% | 0% | 0% | 3% | 100% | | Claude Sonnet 5.5 (top-priced · 1 run per item) | 73% (56–86) | 20% | 7% | 0% | 0% | 0% | 93% | | Claude Haiku 5.5 (mid-priced · 2 runs per item) | 65% (47–79) | 17% | 18% | 0% | 0% | 0% | 85% | | GPT-6.1 Terra (mid-priced · 2 runs per item) | 67% (49–81) | 33% | 0% | 0% | 0% | 3% | 97% | | DeepSeek V4.1 Flash (cheap and free · 3 runs per item) | 61% (43–76) | 14% | 18% | 0% | 7% | 0% | 93% | | Gemini 3.8 Flash (mid-priced · 2 runs per item) | 92% (76–97) | 0% | 0% | 8% | 0% | 50% | 47% | | GLM 5.3 Flash (cheap and free · 3 runs per item) | 40% (25–58) | 31% | 14% | 0% | 14% | 0% | 33% | | Gemini 3.1 Pro (top-priced · 1 run per item) | 70% (52–83) | 0% | 3% | 27% | 0% | 68% | 30% | | Llama 4 Maverick (cheap and free · 3 runs per item) | 3% (1–17) | 16% | 81% | 0% | 0% | 2% | 30% | | Mistral Small (2603) (cheap and free · 3 runs per item) | 1% (0–13) | 1% | 91% | 7% | 0% | 1% | 22% | | gpt-oss-120b (cheap and free · 3 runs per item) | 3% (1–17) | 0% | 93% | 3% | 0% | 5% | 20% | The outcome columns are shares of the 30 files with a planted mistake and add to 100%. False alarms are out of all 40 files that were fine (30 clean twins plus 10 odd-but-valid files). “Clean twins answered right” is out of the 30 clean twins only: the two columns use different denominators. Sorted by caught minus false alarms, which is what “best” means here. Bars show the score; the shaded band is the 95% interval: with only 30 files, the true rate likely sits somewhere in that range. Price group (top-priced, mid-priced, cheap and free) is by price per token (a token is about three-quarters of a word). “Runs per item”: how many times each file was asked. GPT-6 Luna isn’t in this table; see the box below. A separate result, not ranked with the models above #### GPT-6 Luna through Codex (no code or web): caught half - **Model**: GPT-6 Luna, run in OpenAI’s Codex app on my own ChatGPT subscription - **Questions**: the same 30 files with a planted mistake and 40 fine ones; each answered as its own chat, with no code, calculator or web - **Settings**: medium effort, no length limit, one run per file, in small batches - **Graded**: the same answer key, the same code and the same judge as above - **Why apart**: the 11 models above got one message each through the API, at low effort with a 4,000-token limit. Different setups, so I don’t put them in one ranking. **Result.** It recognized 60% of the planted mistakes and stopped and asked on 50%, so it caught 50% (15 of 30). It noticed the mistake and answered anyway on 10%, and missed it and answered anyway on 40%. It refused 10% of the fine files, all 4 times because it said it didn’t have this year’s rules. On the clean twins (the same files with the mistake fixed) it gave the right number 43% of the time. **By kind of mistake.** It caught every pair of conflicting facts (100%) and 83% of swapped digits, but only 33% of impossible ages, 33% of impossible spending, and 0% of amounts over a legal limit. The 14-year-old retiree again The same client file as the example at the top. GPT-6 Luna made the same mistake gpt-oss-120b did: > “Priya is only 14 in 2026, so age-based required minimum distributions do not apply to her traditional IRA. … Her RMD is therefore zero. ANSWER: 0” A required minimum distribution (RMD) is the withdrawal the IRS requires each year from a set age. It noticed she was 14, treated that as a fact, and answered. Which mistakes slip through ### A believable amount over a legal limit was caught least often Pooling every model’s replies, facts that conflict were caught 60% of the time; amounts over a legal limit only 17%. That’s a number that looks normal until you know the rule, like a charitable gift from an IRA above this year’s cap. GPT-6 Luna, tested separately, caught none of them (0%). Not every model followed the pattern: GPT-6.1 Sol and GPT-6.1 Terra missed impossible spending more often than over-limit amounts. Sol, for example, caught 67% of over-limit amounts but only 33% of the impossible spending. **By kind of mistake, every model (for the detail-minded)** | Model | Two digits swapped | An impossible age | Spending that can’t fit | Two facts that conflict | An amount over a legal limit | |---|---|---|---|---|---| | Claude Sonnet 5.5 (top-priced · 1 run per item) | 83% | 100% | 100% | 67% | 17% | | GPT-6.1 Sol (top-priced · 1 run per item) | 100% | 100% | 33% | 100% | 67% | | Gemini 3.1 Pro (top-priced · 1 run per item) | 67% | 83% | 100% | 100% | 0% | | GPT-6.1 Terra (mid-priced · 2 runs per item) | 100% | 100% | 0% | 100% | 33% | | Gemini 3.8 Flash (mid-priced · 2 runs per item) | 83% | 100% | 100% | 100% | 75% | | Claude Haiku 5.5 (mid-priced · 2 runs per item) | 50% | 83% | 92% | 100% | 0% | | DeepSeek V4.1 Flash (cheap and free · 3 runs per item) | 44% | 44% | 89% | 100% | 28% | | GLM 5.3 Flash (cheap and free · 3 runs per item) | 44% | 50% | 33% | 67% | 6% | | gpt-oss-120b (cheap and free · 3 runs per item) | 17% | 0% | 0% | 0% | 0% | | Llama 4 Maverick (cheap and free · 3 runs per item) | 0% | 0% | 0% | 17% | 0% | | Mistral Small (2603) (cheap and free · 3 runs per item) | 0% | 0% | 0% | 6% | 0% | | All 11 models together (every reply, pooled) | 43% | 47% | 41% | 60% | 17% | Six files of each kind. With so few, one reply moves a model’s number a lot; the pooled row is the steadier one. How it fails, in their own words ### Real replies Short excerpts from saved replies. The clients are made up. Missed it, answered anyway The file gave spending of $66,500 a month for a couple with $70,200 a year of income and $675,000 in total. > “Their annual spending is $66,500 * 12 = $798,000. Since their income ($70,200) exceeds their spending ($798,000), they do not need to withdraw anything from the IRA to cover the gap. ANSWER: 0” Mistral Small. It did the multiplication, then compared the numbers backwards. Recognized it, answered anyway > “They would need $620,400 from the IRA to cover the gap, but its $400,000 balance cannot fund that withdrawal. Confirm that the spending figure is monthly, not annual. ANSWER: 620400” GPT-6.1 Sol. Right about the problem, and it still handed back a number that a planning program would store. Knew the rule, didn’t check the number A $140,000 gift to charity straight from an IRA, over this year’s $111,000 limit. The tax rules call this gift a qualified charitable distribution, or **QCD**: > “This assumes the QCD is fully eligible, since it is under the $111,000 2026 limit…” Claude Sonnet 5.5. It knew the limit and never compared the amount to it. What this means for you ### Check what you type in - **Check the quiet numbers most.** A normal-looking amount that breaks a rule slipped through most often (above). - **Read the reply for a warning.** Several models mentioned the problem and answered anyway. If an AI says “confirm this figure”, stop and confirm it before using the number. Limits ### What this test can’t tell you - **Small.** 30 planted mistakes, six of each kind. Intervals are wide; read the gap between “catches most” and “catches almost none”, not the exact order inside a group. - **GPT-6 Luna is one model, one run per file**, run through Codex in a different setup from the others, so its result stands on its own and describes that setup, not every way of using ChatGPT. - **The mistakes are big.** A 14-year-old retiree, monthly spending ten times too high. A birth year off by one, or spending 20% too high, wasn’t tested and is surely harder. - **The easy way out was offered.** Every file said “you may say you need more information”. Without that line, catch rates would likely be lower. I didn’t test that. - **A model judged whether the mistake was named**, and no person has yet checked a sample of those labels. Details below. **How this was graded** Whether a reply ended in a number or in “I need more information” is read by code. Whether it named the planted mistake is judged by another AI model, Claude Haiku 5.5, which sees the reply and the planted mistake but not which model wrote the reply. A second model from another lab, GPT-6 Luna, judged 176 of the replies again; the two agreed on whether the mistake was named 97.7% of the time. That is two models agreeing, not a measured error rate. I also had the AI agent leading this project (Claude, not a person) re-read every reply that a simple word-and-number filter marked as possibly naming the mistake, and applied the judge’s own written rule, the same for every model. It changed 18 labels, all from “didn’t name it” to “named it” (for example, a reply that spelled out the excess over a limit but had been scored as missing it), and 0 the other way. The scores on this page include those corrections; the list of every change, with the quote behind it, is in the project’s files. A person checking a random sample of labels is still owed. “Missed it, answered anyway” depends on that judgment too: it means the reply gave a number and the judge found no mention of the mistake. **For the technical reader: how this was run** 11 models through OpenRouter, each pinned to one host with data collection off. One user message, no system prompt, reasoning effort low, a reply limit of 4,000 tokens. The budget cap set the runs per file: three for cheap models, two for middle-priced, one for the most expensive; a repeated file counts once, as the share of its runs that caught the mistake. Intervals are 95% Wilson intervals over files. These settings differ from the first test (October: high reasoning effort, long replies), so don’t compare the two directly. GPT-6 Luna through Codex: Answered in agent sessions of 5-11 prompts each (earlier prompts stayed in the conversation), told to treat each as its own chat, no code, web or files; no token limit. The 11 API models got one message per question at low effort with a 4,000-token limit. The first attempt was thrown out because the agent answering wrote bare final numbers instead of answering each question the way a chat would; that attempt is archived and not scored. Same graders and judge. The lead agent re-read every reply the judge marked as not naming the mistake and changed 0 labels; the reverse check (replies marked as naming it) hasn’t been done, and a person’s spot-check is still owed. [Read the full report →](https://plannerbench.org/report.html) [Back to **Results: the numbers behind each finding**](https://plannerbench.org/results.html) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## Example questions Source: https://plannerbench.org/questions.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt Here is one real question of each of the 14 kinds I asked, with the right answer. Under each one is how many questions of that kind each model got right in my first test, in October 2026. Each tax question also told the model to use 2026 federal tax law, including the new $6,000 senior deduction, and every question asked for the answer on a line of its own. I left those parts out here so the questions are easier to read. The totals differ because I asked some models more questions than others, and a few replies never came back. Claude Haiku 5.5 is listed twice because I asked it two separate ways. [The first test in full →](https://plannerbench.org/alone.html#october) ### One rule One rule at a time: a tax bracket, an RMD divisor, a Social Security claiming factor, a Medicare premium tier. #### Federal income tax for one year In 2026, a single filer (age 65 or older) has $191,000 of wages, $34,000 of traditional IRA distributions, $0 of long-term capital gains and qualified dividends, and $41,400 of Social Security benefits. They take the standard deduction and have no credits. What is their total federal income tax including any Net Investment Income Tax, rounded to the nearest dollar? Right answer: **$53,909** Right, on questions of this kind - Claude Sonnet 5.5 **10 of 10** - Claude Haiku 5.5 **10 of 10** - DeepSeek V4.1 Flash **10 of 15** - Claude Haiku 5.5 (second way) **15 of 15** - GLM 5.3 Flash **9 of 15** - Qwen3.8 Flash **0 of 15** - gpt-oss-120b **0 of 15** #### How much of a Social Security benefit is taxable Using the IRS Publication 915 worksheet, a single filer receives $24,500 of Social Security benefits and has $65,000 of other income (no tax-exempt interest, no adjustments). How much of the Social Security is taxable, rounded to the nearest dollar? Right answer: **$20,825** Right, on questions of this kind - Claude Sonnet 5.5 **10 of 10** - Claude Haiku 5.5 **10 of 10** - DeepSeek V4.1 Flash **15 of 15** - Claude Haiku 5.5 (second way) **14 of 15** - GLM 5.3 Flash **11 of 15** - Qwen3.8 Flash **12 of 15** - gpt-oss-120b **7 of 15** #### This year’s required IRA withdrawal Under SECURE 2.0, a traditional IRA owner born in 1949 turns 77 in 2026. The IRA balance on December 31, 2025 was $1,411,000. Their spouse is not more than 10 years younger. What is the required minimum distribution for 2026, rounded to the nearest dollar? Answer 0 if no RMD is required for 2026. Right answer: **$61,616** Right, on questions of this kind - Claude Sonnet 5.5 **10 of 10** - Claude Haiku 5.5 **10 of 10** - DeepSeek V4.1 Flash **15 of 15** - Claude Haiku 5.5 (second way) **15 of 15** - GLM 5.3 Flash **11 of 15** - Qwen3.8 Flash **6 of 15** - gpt-oss-120b **5 of 15** #### The monthly benefit when Social Security starts early A worker born in 1965 has a primary insurance amount (PIA) of $1,530/month. They start retirement benefits at age 63 and 3 months. Ignoring COLAs and the earnings test, what is their monthly benefit, rounded to the nearest dollar? Right answer: **$1,167** Right, on questions of this kind - Claude Sonnet 5.5 **8 of 10** - Claude Haiku 5.5 **9 of 10** - DeepSeek V4.1 Flash **15 of 15** - Claude Haiku 5.5 (second way) **12 of 15** - GLM 5.3 Flash **14 of 15** - Qwen3.8 Flash **14 of 15** - gpt-oss-120b **14 of 15** #### How big a Roth conversion fits in a tax bracket In 2026, a married couple filing jointly under age 65 has $218,000 of ordinary income (no Social Security, no capital gains) and takes the standard deduction. How large a Roth conversion can they do without any of it being taxed above the 22% bracket? Answer 0 if they are already above it. Right answer: **$25,600** Right, on questions of this kind - Claude Sonnet 5.5 **10 of 10** - Claude Haiku 5.5 **10 of 10** - DeepSeek V4.1 Flash **11 of 15** - Claude Haiku 5.5 (second way) **14 of 15** - GLM 5.3 Flash **13 of 15** - Qwen3.8 Flash **7 of 15** - gpt-oss-120b **6 of 15** #### The Medicare Part B premium at a given income In 2026, a Medicare enrollee filing jointly had a 2024 modified adjusted gross income of $342,800. What is their 2026 monthly Medicare Part B premium per person, including any IRMAA surcharge? Give dollars and cents. Right answer: **$527.50** Right, on questions of this kind - Claude Sonnet 5.5 **10 of 10** - Claude Haiku 5.5 **9 of 10** - DeepSeek V4.1 Flash **11 of 15** - Claude Haiku 5.5 (second way) **13 of 15** - GLM 5.3 Flash **3 of 15** - Qwen3.8 Flash **0 of 15** - gpt-oss-120b **0 of 15** ### A few steps Two rules that interact, like a Roth conversion that also makes more Social Security taxable (the “tax torpedo”). #### The tax a Roth conversion adds when it also makes more Social Security taxable In 2026, a married couple filing jointly (both spouses age 65 or older) has $40,000 of traditional IRA distributions, $0 of long-term capital gains and qualified dividends, and $39,900 of Social Security benefits, with no wages. They take the standard deduction and have no credits. If they also convert $14,000 from a traditional IRA to a Roth IRA in 2026, by how many dollars does their total federal income tax (including any Net Investment Income Tax) go up? Round to the nearest dollar. Right answer: **$2,853** Right, on questions of this kind - Claude Sonnet 5.5 **10 of 10** - Claude Haiku 5.5 **9 of 10** - DeepSeek V4.1 Flash **14 of 15** - Claude Haiku 5.5 (second way) **12 of 15** - GLM 5.3 Flash **11 of 15** - Qwen3.8 Flash **1 of 15** - gpt-oss-120b **3 of 15** #### How much more tax a surviving spouse owes A married couple, both age 65 or older, files jointly. They have $58,500 of traditional IRA distributions, $20,500 of long-term capital gains and qualified dividends, and Social Security benefits of $24,500 (spouse A) and $22,500 (spouse B), with no wages. Spouse B dies. In the survivor's first full year filing as single (no dependents), spouse A keeps the larger of the two Social Security benefits and has the same IRA distributions and gains. Using 2026 law and amounts for both years, standard deduction and no credits, how many dollars more total federal income tax (including any Net Investment Income Tax) does the survivor owe than the couple did? Give a negative number if the survivor owes less. Round to the nearest dollar. Right answer: **$4,635** Right, on questions of this kind - Claude Sonnet 5.5 **8 of 10** - Claude Haiku 5.5 **6 of 10** - DeepSeek V4.1 Flash **7 of 15** - Claude Haiku 5.5 (second way) **7 of 15** - GLM 5.3 Flash **4 of 15** - Qwen3.8 Flash **0 of 15** - gpt-oss-120b **0 of 15** ### Several rules at once Tax, Social Security, the senior deduction and Medicare’s income-based premiums (IRMAA) meeting in one household. #### Tax for retirees whose income shrinks the new senior deduction In 2026, a married couple filing jointly (both spouses age 65 or older) has $124,000 of traditional IRA distributions, $37,500 of long-term capital gains and qualified dividends, and $54,300 of Social Security benefits, with no wages. They take the standard deduction and have no credits. What is their total federal income tax including any Net Investment Income Tax, rounded to the nearest dollar? Right answer: **$23,555** Right, on questions of this kind - Claude Sonnet 5.5 **1 of 10** - Claude Haiku 5.5 **0 of 10** - DeepSeek V4.1 Flash **3 of 15** - Claude Haiku 5.5 (second way) **0 of 15** - GLM 5.3 Flash **4 of 15** - Qwen3.8 Flash **0 of 1** - gpt-oss-120b **1 of 15** #### What a Roth conversion costs in tax plus Medicare premiums In 2026, a married couple filing jointly (both spouses age 65 or older, both enrolled in Medicare Part B) has $291,000 of traditional IRA distributions, $0 of long-term capital gains and qualified dividends, and $71,500 of Social Security benefits, with no wages, standard deduction and no credits. They are considering converting $40,000 from a traditional IRA to a Roth IRA in 2026. Medicare sets Part B premiums from the tax return two years earlier, so 2026 income sets 2028 premiums; assume the 2026 premiums and tiers still apply in 2028 and that nothing else changes. What is the total added cost of the conversion: the increase in 2026 federal income tax (including any Net Investment Income Tax) plus the increase in their combined 2028 Part B premiums for the full year? Round to the nearest dollar. Right answer: **$9,600** Right, on questions of this kind - Claude Sonnet 5.5 **8 of 10** - Claude Haiku 5.5 **3 of 10** - DeepSeek V4.1 Flash **8 of 15** - Claude Haiku 5.5 (second way) **6 of 15** - GLM 5.3 Flash **5 of 15** - Qwen3.8 Flash **0 of 5** - gpt-oss-120b **1 of 15** #### What a surviving spouse pays in added tax plus Medicare premiums A married couple, both age 65 or older and both enrolled in Medicare Part B, files jointly. They have $155,500 of traditional IRA distributions, $16,500 of long-term capital gains and qualified dividends, and Social Security benefits of $26,500 (spouse A) and $11,200 (spouse B), with no wages. Spouse B dies. In the survivor's first full year filing as single (no dependents), spouse A keeps the larger of the two Social Security benefits and has the same IRA distributions and gains. Using 2026 law and amounts for both years, standard deduction and no credits, and assuming each year's Part B premiums are set by that same year's income, how many dollars more does the survivor pay per year in federal income tax (including any Net Investment Income Tax) plus Part B premiums than the couple paid together? Give a negative number if the survivor pays less. Round to the nearest dollar. Right answer: **$10,768** Right, on questions of this kind - Claude Sonnet 5.5 **2 of 10** - Claude Haiku 5.5 **2 of 10** - DeepSeek V4.1 Flash **5 of 15** - Claude Haiku 5.5 (second way) **3 of 15** - GLM 5.3 Flash **3 of 15** - Qwen3.8 Flash **0 of 11** - gpt-oss-120b **0 of 7** ### Compliance review A planted problem a reviewer must catch: a rollover issue, a missed sales-charge discount, a misleading sentence. #### Spotting misleading sentences in a client email FINRA Rule 2210(d)(1) says retail communications may not contain any false, exaggerated, unwarranted, promissory or misleading statement or claim, and may not predict or project performance. A representative drafted this email to retail clients about a stock mutual fund: 1. The fund is not guaranteed and can lose value, so you could lose money. 2. This fund is guaranteed to outperform the stock market over the next five years. 3. Past performance does not guarantee future results. 4. This is a can't-miss opportunity that will double your money. 5. CDs are FDIC-insured up to $250,000 per depositor; mutual fund shares are not FDIC-insured. 6. The fund's expense ratio is 0.65% a year, which is taken out of fund assets. Which numbered sentences violate Rule 2210(d)(1)? Right answer: **Sentences 2, 4** Right, on questions of this kind - Claude Sonnet 5.5 **10 of 10** - Claude Haiku 5.5 **10 of 10** - DeepSeek V4.1 Flash **15 of 15** - Claude Haiku 5.5 (second way) **15 of 15** - GLM 5.3 Flash **15 of 15** - Qwen3.8 Flash **15 of 15** - gpt-oss-120b **15 of 15** #### What a 401(k) rollover recommendation should have weighed You are a supervisor reviewing a registered representative's rollover recommendation under Regulation Best Interest's Care Obligation and the factors in FINRA Regulatory Notice 13-45. The client is 57 and left their employer at age 52. They plan to withdraw $45,000 a year starting now to cover expenses until they claim Social Security. Their 401(k) is worth $1,200,000, including $352,000 of the former employer's stock with a cost basis of $46,000. The 401(k)'s funds charge 0.15% a year in total. The recommendation is to roll the whole 401(k), employer stock included, into an IRA invested in funds charging 0.95% a year plus a 1% annual advisory fee. Which of these issues apply to this client and should have been weighed before recommending the rollover? - AGE55: The client left the employer in or after the year they turned 55, is under 59½, and plans withdrawals before 59½. Withdrawals from the 401(k) would avoid the 10% early-withdrawal penalty, but IRA withdrawals would not. - NUA: Rolling low-basis employer stock into an IRA gives up net unrealized appreciation tax treatment. - COST: The IRA would cost the client more each year than staying in the plan. Answer with every code that applies, separated by commas, or NONE. Right answer: **COST, NUA** Right, on questions of this kind - Claude Sonnet 5.5 **10 of 10** - Claude Haiku 5.5 **10 of 10** - DeepSeek V4.1 Flash **13 of 15** - Claude Haiku 5.5 (second way) **12 of 15** - GLM 5.3 Flash **12 of 15** - Qwen3.8 Flash **11 of 15** - gpt-oss-120b **11 of 15** #### The sales-charge discount on a mutual fund purchase A client is buying $87,000 of Class A shares of a mutual fund. The prospectus sales charge, as a percentage of the offering price, is: less than $50,000: 5.75%; $50,000 to less than $100,000: 4.50%; $100,000 to less than $250,000: 3.50%; $250,000 to less than $500,000: 2.50%; $500,000 to less than $1,000,000: 2.00%; $1,000,000 or more: 0%. Under the prospectus's rights of accumulation, the purchase is combined with the current value of holdings in the same fund family owned by the client, the client's spouse, and their children under age 21, in any type of account. Current holdings: the client's IRA in the same fund family, $0; the spouse's taxable account in the same fund family, $90,000; their 14-year-old child's account in the same fund family, $4,000; their 24-year-old child's account in the same fund family, $31,000; the client's brokerage holdings in a different fund family, $138,000. What sales charge percentage should this purchase receive? Right answer: **3.5%** Right, on questions of this kind - Claude Sonnet 5.5 **10 of 10** - Claude Haiku 5.5 **10 of 10** - DeepSeek V4.1 Flash **15 of 15** - Claude Haiku 5.5 (second way) **15 of 15** - GLM 5.3 Flash **15 of 15** - Qwen3.8 Flash **15 of 15** - gpt-oss-120b **15 of 15** [Home →](https://plannerbench.org/) [Results →](https://plannerbench.org/results.html) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## A private demo: the same calculator, inside Claude and ChatGPT. Source: https://plannerbench.org/chat.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt Demo walkthrough · private connector, not public What it is This is a private demo; you can’t install it. The calculator from [With a calculator](https://plannerbench.org/calculator.html) also works as an add-on (a “connector”) for Claude and ChatGPT. Someone asks a retirement question in everyday words. The calculator, not the chat AI, decides which fact to ask for next, does the calculations, and says what it assumed. The chat AI is instructed to be friendly and quote the calculator’s numbers exactly. Can I use it? No. It runs at a private link while I test it, and there’s no public install. This page shows what it does, using its real replies to a made-up couple. A conversation, from the calculator’s side ### What it sends back, word for word The chat AI turns these replies into conversation. Its instructions say to quote these numbers and never work out its own; in testing, a checker flags any number in a conversation that no calculator reply contains. 1. 1 · Someone asks, with no details yet “We’re both retired. Should we be converting some of our IRA to a Roth, and how much?” It asks one question, says why, and what to have handy How old are you and your spouse this year? Why it matters: Age decides when required withdrawals start, Medicare, and how long the plan runs; being married decides the tax brackets, and a widow or widower may have survivor benefits and a spouse's options on inherited accounts. - Birth dates for both of you - Social Security statements from ssa.gov for both of you - The latest statement for every account (IRAs, 401(k)s, Roth IRAs, brokerage, savings), with what you paid (cost basis) for brokerage accounts - Paychecks or yearly wages, and any pension or annuity amounts - Your last two tax returns (they set Medicare premiums, and Form 8606 shows any after-tax IRA money) - Roughly what you spend in a year 2. 2 · A few questions later, every fact in The couple has answered ages, accounts, Social Security, spending and state, one question at a time. The answer, in sentences Convert about $56,000 a year from Dana's IRA, 2026 through 2035. By 2055 that leaves the family about $132,193 better off after taxes, in today's money, than not converting. - **Best plan**: $56,000 a year for 10 years (2026–2035) - **Total converted**: $560,000 - **Better off by 2055 (today's money)**: +$132,193 - **Change in lifetime tax (today's money)**: −$113,779 - **Change in lifetime Medicare premiums (today's money)**: +$0 What it costs now - Income tax on the extra money taken out or converted **$62,206** IRC §1(j); Rev. Proc. 2025-32 - Arizona income tax (2.5%) **$11,422** A.R.S. §43-1011; Arizona Form 140 - Dividends and gains lose their 0% tax rate as income rises **$9,503** IRC §1(h); Qualified Dividends and Capital Gain Tax Worksheet - More of their Social Security becomes taxable as other income rises (the "tax torpedo") **$8,246** IRC §86; IRS Pub 915 Worksheet 1 - Total, 2026–2035 (the biggest reasons are listed; smaller ones make up the rest) **$91,434** What it saves later - Less income tax later, because the IRA is smaller and so are the required withdrawals **$247,329** IRC §1(j) - Less Arizona income tax **$33,371** A.R.S. §43-1011; Arizona Form 140 - More dividends and gains stay at the 0% rate **$29,207** IRC §1(h); Qualified Dividends and Capital Gain Tax Worksheet - Less of the 3.8% tax on investment income **$21,835** IRC §1411; Form 8960 - Total, 2036–2055 **$331,742** How these fit the figures above: the costs and savings are each year’s own dollars, added up as they would appear on the tax returns. The lifetime tax change (−$113,779) and “better off” (+$132,193) are in today’s money, which counts a dollar paid decades from now as worth less. “Better off” also counts what’s left in every account at the end, so it isn’t the tax change alone. The two kinds of figure measure different things and aren’t meant to add up. Said out loud: what it assumed - Assumed: Dana on Medicare: Yes, from 65, because nearly everyone enrolls at 65. - Assumed: Lee on Medicare: Yes, from 65, because nearly everyone enrolls at 65. - Assumed: The plan runs through: 2055, because until the younger one would be 95. - Assumed: Investment return: 5% a year on every account, because a planning assumption, steady every year. - Assumed: Inflation: 2.5% a year, because tax brackets and Medicare lines rise with it after 2026. And what could change the answer With a 7% return, the better choice becomes $100,000 a year for 10 years (about $85,666 better). 24 facts said, 5 assumed, 1 worked out · 1,402 plans run for this one answer, 1.4 s on a laptop 3. 3 · Someone types a Social Security number “My SSN is 000-12-3456, can you check my Social Security?” Turned away before it reaches any math That looks like a Social Security, account or ID number. The engine doesn't need it, so please leave it out and send the rest again. Nothing you sent was kept. How it behaves ### Most rules live in the calculator, not in a prompt - **It picks the next question.** It reruns the plan with each unknown fact both ways and asks only for the one that could change the choice or move the answer more than 2%. You can skip any question; it then says what it assumed instead. - **Numbers come from the calculator.** The chat AI is instructed to quote only the calculator’s numbers. In tests, a checker lists any number in a conversation that no calculator reply contains. That catches a slip; it can’t make one impossible. - **Close calls get both plans.** When two plans land within $1,000 over 25 years, it says “it’s close” instead of naming a winner. - **It says what it doesn’t cover.** A question outside its rules gets “that isn’t covered yet”, not a guess. - **It keeps nothing.** No database, no files, nothing you type in its logs. Anything that looks like a Social Security, account or ID number is refused and never repeated back. - **Education, not advice.** It prices choices; it never picks investments or products. How it’s checked ### Tested before anyone used it, then fixed from real use 69 / 69 #### Simulated clients For each household question with a fact left out, a scripted client answers only what the calculator asks. A pass means the same decision as with every fact known, the number within 2%, an answer within 6 questions, the hidden fact asked for, and no wasted questions. No AI model is involved. 6 + 17 notes #### Test drives become tests I used it in Claude the way a client would, with hard households (a widow with inherited IRAs, someone still working past 62), and wrote down every rough edge. Each was fixed and the session saved as a test so it can’t come back. One changed a headline: the Social Security earnings limit erased a “claim at 62” gain. 210 #### Automated tests Every rule, the everyday-words input, the ID-number refusal, and a full round trip over the network. Given the same households in everyday words, it reproduces the household decisions’ answer key. **For the technical reader: how it’s built** - **An MCP server** (the Model Context Protocol, the standard way to plug tools into AI models). One server, two ways in: a local process for Claude Code and the benchmark runners, and a private web address for Claude and ChatGPT. The replies above are from engine commit `3b7d8bc`. - **Tools.** `plan_check` (the front door, in everyday terms) plus eight calculators for builders: federal income tax, taxable social security, required minimum distribution, social security benefit, medicare part b premium, tax tables, run scenario, solve roth conversion. - **Replies** are one of `need_more` (one question), `answer`, `not_covered` or `rejected`, each with a `how_to_reply` the chat model follows. - **Stateless over HTTP** (Streamable HTTP, a fresh server per request): the conversation carries the facts, so there is nothing to store or leak. Requests capped at 64 KB, every list capped, a per-address rate limit. - **A card inside the chat** (MCP Apps): the answer, the conversion curve and what it costs vs. saves. The text reply is always the whole answer, for apps that don’t draw cards. - **The benchmark sees the same engine.** PlannerBench calls the builder tools over the local process, so a score on this site and an answer in the chat come from the same code. [Next **Replays: every calculator call, step by step**](https://plannerbench.org/replay.html) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## Can you trust AI with retirement math? Source: https://plannerbench.org/report.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt PlannerBench · full report Peyton Thalman · updated 2026-11-15 · two rounds of tests (October and November 2026), then an independent audit ### Summary I'm a financial advisor. More and more of the people I meet get their financial advice from ChatGPT, so I tested how often AI gets retirement math right, against an answer key I could prove. 1. **Simple questions: almost always right.** On one-rule questions (this year's tax bracket, an IRA withdrawal), Claude Sonnet 5.5 and Claude Haiku 5.5 both scored 97% in the first test. 2. **Real-life questions: it depends which AI.** When taxes, Social Security and Medicare meet in one household, scores spread out. In the first test, Sonnet got 37% of these and Haiku 17%, and models from several labs made the same misreading of a rule new in 2025. In the newer, harder set of 42 questions (11 models, low reasoning effort), GPT-6.1 Sol got 98%, Claude Sonnet 5.5 62%, and all five cheap models under 30%. 3. **A calculator helps, if it shows its work, and only for some models.** Handed only the right number, Claude Haiku overruled it with its own wrong rule (40% → 62% on the 50 hardest first-test questions). When the calculator also showed the rule and the IRS form line, Haiku got 100%, and 96% on 50 questions held back. In the newer set, DeepSeek V4.1 Flash went from 29% to 79% with the calculator; three other cheap models called it and scored 10% or less, and the best model gained nothing. 4. **Tell it something wrong: GPT-6.1 Sol and Claude Sonnet 5.5 catch most of it, three cheap models almost none. This is the biggest open gap.** In 30 client files with one planted mistake each, GPT-6.1 Sol caught 80% and Claude Sonnet 5.5 73%, with almost no false alarms. Three cheap models caught 1–3% and gave a confident number 81–93% of the time. **And ChatGPT?** GPT-6 Luna on a ChatGPT subscription got 71% of one-rule questions but 17% of the hardest ones, and caught half the planted mistakes. It was a different setup, so it is reported in its own section below. **The plain answer.** For one well-known rule, the best AIs are reliable. For a whole household, it depends heavily on which AI you ask: the cheap and free models I tested, at quick-answer settings, got most of the hard questions wrong. And the answer is only as good as what you type in: in my tests, three cheap models almost never questioned an impossible fact, and even the best missed about one in four. Check your inputs, and take big decisions to a professional. **How this was built.** I directed AI coding agents (mostly Claude Code) and made the calls: what to test, what counts as right, what to spend, what to publish. Because AI did most of the work, the checks are set up so no agent is the only grader of its own work (the lead agent did re-check 18 judge labels; that is disclosed below, and a human spot-check is owed), and a model from another lab audited every published claim on 2026-11-15. It found 12 errors. They are fixed in this version and listed under [Corrections](https://plannerbench.org/report.html#corrections). **Chart: Catch rate against false-alarm rate** ([image](https://plannerbench.org/charts/catch_vs_false_alarm.svg)) | Item | Value | |---|---| | Gemini 3.8 Flash | caught 92%, false alarms 50% | | GPT-6.1 Sol | caught 80%, false alarms 3% | | Claude Sonnet 5.5 | caught 73%, false alarms 0% | | Gemini 3.1 Pro | caught 70%, false alarms 68% | | GPT-6.1 Terra | caught 67%, false alarms 3% | | Claude Haiku 5.5 | caught 65%, false alarms 0% | | DeepSeek V4.1 Flash | caught 61%, false alarms 0% | | GLM 5.3 Flash | caught 40%, false alarms 0% | | gpt-oss-120b | caught 3%, false alarms 5% | | Llama 4 Maverick | caught 3%, false alarms 2% | | Mistral Small (2603) | caught 1%, false alarms 1% | ### Why I made this I meet most people by knocking on their door. When someone tells me ChatGPT answered their retirement question, I ask how they know it's right. Usually they Googled it, and the top result was another AI's answer. I wanted to know, with evidence, which planning questions a model can be trusted with on its own, and which need verified math behind it. ### What I built - **An answer key I could prove.** The 2026 federal tax, Social Security, required-withdrawal and Medicare rules written as a program: the planning engine. A second engine, written by an agent that never saw the first, from the published rules alone, agrees with it on 4,000 of 4,000 random households within $1. PolicyEngine-US, an open-source tax model nobody on this project wrote, agrees on 1,600 of 1,600 first-test tax answers within $2. - **The first test (October 2026).** 2,800 public questions with one exact answer each (plus 700 held back), from one-rule questions to ones where several rules meet. Seven model setups from five labs, reasoning effort high, 16,000–32,000-token replies. - **The calculator.** The engine as tools a model can call while it answers (an MCP server), and as a connector in Claude and ChatGPT. - **The newer, harder set (November 2026).** My own audit of the first test ([AUDIT.md](https://github.com/peyton-thalman/plannerbench/blob/v2/AUDIT.md)) found two problems: the top-scoring models had nearly maxed out the simple questions, and every fact I gave them was true. So I added 42 harder questions and 30 client files with a planted mistake, and ran 11 models at low reasoning effort with 4,000-token replies, for $4.51 against a $5 cap. The two rounds used different settings, so their scores are reported separately and never compared directly. ### Findings 1 and 2: AI on its own #### The first test (October) | Model | One rule | Several rules at once | Overall | |---|---|---|---| | Claude Sonnet 5.5 | 97% | 37% | 84% | | Claude Haiku 5.5 | 97% | 17% | 77% | Five more setups (DeepSeek V4.1 Flash, GLM 5.3 Flash, gpt-oss-120b, Qwen3.8 Flash, Haiku through OpenRouter) are on the site's "AI on its own" page with 95% intervals; 10–15 questions per task per model. **One rule explains a large share of the misses.** The 2025 tax law gave each person 65 or older an extra $6,000 deduction that shrinks by 6% of income over $150,000 on a joint return, worked per person. Models from several labs applied the phase-out to the couple as a whole and gave the same wrong answer. Across the two published first-test runs, this one misreading accounts for 99 of 342 wrong answers (29%), and 99 of the 139 wrong answers I could trace to a named misreading (71%). Pasting the IRS's two-line instructions in front of the question fixed most of them. #### The newer, harder set (November) Each of the 42 questions makes two or more rules meet: the largest Roth conversion that leaves Medicare premiums alone two years later; a conversion's cost in federal plus Arizona tax; taxable Social Security with muni interest and a charitable gift from an IRA (a QCD); an inherited IRA's 2026 required withdrawal under the 10-year rule; tax when part of a required withdrawal goes to charity; a widow's first year; Arizona tax for retirees. Each key comes from the planning engine and must match the second engine to the dollar. Sources and examples: [v2/data/HARD.md](https://github.com/peyton-thalman/plannerbench/blob/v2/v2/data/HARD.md). | Model | Maker | Alone | Ran out of room | $ per right answer | |---|---|---|---|---| | GPT-6.1 Sol | OpenAI | 98% [88–100] | 0% | $0.0032 | | GPT-6.1 Terra | OpenAI | 80% [65–89] | 0% | $0.0019 | | Claude Sonnet 5.5 | Anthropic | 62% [47–75] | 0% | $0.0173 | | Claude Haiku 5.5 | Anthropic | 52% [38–67] | 2% | $0.0022 | | Gemini 3.1 Pro | Google | 29% [17–44] | 0% | $0.0513 | | Gemini 3.8 Flash | Google | 29% [17–44] | 36% | $0.0195 | | DeepSeek V4.1 Flash | DeepSeek (open) | 29% [17–44] | 71% | $0.0036 | | GLM 5.3 Flash | Z.ai (open) | 15% [8–29] | 20% | $0.0063 | | gpt-oss-120b | OpenAI (open) | 7% [2–19] | 0% | $0.0016 | | Mistral Small | Mistral (open) | 7% [2–19] | 0% | $0.0087 | | Llama 4 Maverick | Meta (open) | 6% [2–17] | 0% | $0.0185 | - **Only one model is near the ceiling.** GPT-6.1 Sol missed one of 42 at low effort. Everything else is between 6% and 80%. The Arizona family is the hardest: only Sol got any of it right alone. - **Running out of room is a big part of the cheap models' scores.** DeepSeek hit the 4,000-token limit on 71% of questions and Gemini 3.8 Flash on 36%. Scores at a higher setting may be higher; I didn't run that comparison. - Haiku's 52% averages its two runs; its first run alone was 20 of 42 (48%), the baseline used in the paired tool comparison below. ### Finding 3: with a calculator #### Hand it the number and it argues; show it the rule and it listens (October) Claude Haiku 5.5, effort high, on the 50 hardest first-test questions: | | Alone | Calculator 1.0 (number only) | Calculator 1.1 (number, rule, form line) | |---|---|---|---| | Public questions | 40% (20/50) | 62% (31/50) | 100% (50/50) | | Held-back questions | 44% | | 96% (48/50) | With calculator 1.0, most misses were Haiku calling the tool, getting the right answer, and replacing it with its own per-couple reading of the senior deduction, in its own words ("the tool returned $16,074, but I believe it overstates the senior deduction phase-out"). The engine's answers didn't change between 1.0 and 1.1; only the reply did. Both held-back misses were Haiku overruling the tool again. This was tested on one model; it shows that what a tool says back matters, not that it works for every model. **Whole-household decisions.** Each question is a made-up household of about 16 facts and a real choice (how much to convert to a Roth, when to claim Social Security, how to give, how to empty an inherited IRA). One question from each of 10 families plus a trap, one try each: | Model | Right decision and every number within 2%, alone | With the calculator | |---|---|---| | Claude Haiku 5.5 | 4/11 | 11/11 | | Claude Sonnet 5.5 | 7/10 (3 ran out of room) | 11/11 | | DeepSeek V4.1 Flash | 2/11 | 11/11 | Every key agrees with the second engine on every field of every year, and every 2026 return agrees with PolicyEngine-US (203/203). With one fact left out (a Social Security amount or an IRA balance), DeepSeek with the calculator asked for it 9 times in 10 and got 9 of 10 fully right. Small samples: this shows the gap, not a ranking. #### The newer set: a cheap model catches up, others don't (November) Paired: each model's first run alone against its one run with the calculator, on the same 42 questions. | Model | Alone (first run) | With the calculator | Fixed / broken | Called a tool | |---|---|---|---|---| | GPT-6.1 Sol | 41/42 | 40/42 | 0 / 1 | 79% | | GPT-6.1 Terra | 34/42 | 36/42 | 3 / 1 | 81% | | DeepSeek V4.1 Flash | 12/42 | 33/42 | 21 / 0 | 90% | | Claude Haiku 5.5 | 20/42 | 26/42 | 8 / 2 | 50% | | GLM 5.3 Flash | 7/42 | 18/42 | 11 / 0 | 86% | | gpt-oss-120b | 3/42 | 4/42 | 2 / 1 | 79% | | Mistral Small | 3/42 | 3/42 | 3 / 3 | 79% | | Llama 4 Maverick | 3/42 | 2/42 | 0 / 1 | 79% | Sonnet and the Gemini models were not run with the calculator (cost). The "Alone" column in the earlier table averages two runs for some models; this one uses the first run, the one the calculator run is paired with. - **DeepSeek scored higher than Sonnet alone, at about a fifteenth of the cost per right answer.** DeepSeek with the engine got 33 of 42 right; Sonnet alone got 26. Both were right on 23, only DeepSeek on 10, only Sonnet on 3, neither on 6 (exact McNemar p = 0.09). The difference isn't statistically significant. - **The calculator helped the models that were running out of room.** DeepSeek fixed 21 questions and broke none (paired, p < 0.001); GLM fixed 11 and broke none (p = 0.001). The 4,000-token limit applies to each turn of the tool loop, up to 8 turns: 10 of 336 tool-track replies used more than 4,000 output tokens in total (DeepSeek 9, GLM 1; the most was 7,135). Counting only DeepSeek's replies within 4,000 tokens in total, it still got 27 of 42 (64%). - **Three cheap models called the tools and it didn't matter.** gpt-oss, Llama and Mistral Small called a tool on 79% of questions and scored 10% or less: wrong inputs, or one call where the question needed three. - **Haiku didn't call a tool on half the questions**, and said so ("I did not run this through the calculator tools"). Its gain, 20 to 26 of 42, isn't significant (p = 0.11). - **The best model gained nothing.** Sol went from 41 to 40 of 42. Both misses with tools were Arizona questions where it took the federal numbers from the tool and left out Arizona's $2,100 age-65 exemption. ### Finding 4: wrong facts **Setup.** Each note is a short planner's intake file and one question. 30 notes have one planted mistake each, in five kinds: two digits swapped in a balance; a birth year that makes the client 14 (or 138); monthly spending inflated ten- or twelvefold, so the household would spend more than everything it owns in a year; facts that conflict ("married" next to "filing single"); an amount over a legal limit. Each has a clean twin that differs in only that fact, and 10 odd-but-valid notes (a 90-year-old IRA owner, a contribution at exactly the limit) make sure a model can't win by flagging anything unusual. Every prompt ends the same way: give a number, or end with `ANSWER: NEED_INFO` and say what you need. Examples: [v2/data/BAD_INPUTS.md](https://github.com/peyton-thalman/plannerbench/blob/v2/v2/data/BAD_INPUTS.md). A reply to a corrupted note lands in one of five places: - **Caught it**: named the planted problem and gave no number. - **Recognized it, answered anyway**: named the problem and still ended with a number. - **Missed it, answered anyway**: a confident number with no mention of the problem. - **Declined for another reason**: no number, but not because of the planted problem. - **Ran out of room**: no answer because the reply hit the token limit. Counted as not caught. Whether a reply ended in a number or NEED_INFO is read by a rule. Whether it named the planted problem was judged by Claude Haiku 5.5, which saw the reply and the planted problem but not which model wrote it. A second judge from another lab (GPT-6 Luna) re-judged 176 of the corrupted-note replies and agreed with the first on 97.7% of "named the problem" labels (kappa 0.95). Then the lead AI agent (Claude), not a person, adjudicated every reply the judge marked as not naming the problem (details below). | Model | Caught it | Recognized it, answered anyway | Missed it, answered anyway | Declined, other reason | Ran out of room | False alarms (40 valid notes) | Clean twins right (30) | |---|---|---|---|---|---|---|---| | GPT-6.1 Sol | 80% [63–90] | 20% | 0% | 0% | 0% | 3% | 100% | | Claude Sonnet 5.5 | 73% [56–86] | 20% | 7% | 0% | 0% | 0% | 93% | | GPT-6.1 Terra | 67% [49–81] | 33% | 0% | 0% | 0% | 3% | 97% | | Claude Haiku 5.5 | 65% [47–79] | 17% | 18% | 0% | 0% | 0% | 85% | | DeepSeek V4.1 Flash | 61% [43–76] | 14% | 18% | 0% | 7% | 0% | 93% | | Gemini 3.8 Flash | 92% [76–97] | 0% | 0% | 8% | 0% | 50% | 47% | | Gemini 3.1 Pro | 70% [52–83] | 0% | 3% | 27% | 0% | 68% | 30% | | GLM 5.3 Flash | 40% [25–58] | 31% | 14% | 0% | 14% | 0% | 33% | | Llama 4 Maverick | 3% [1–17] | 16% | 81% | 0% | 0% | 2% | 30% | | gpt-oss-120b | 3% [1–17] | 0% | 93% | 3% | 0% | 5% | 20% | | Mistral Small | 1% [0–13] | 1% | 91% | 7% | 0% | 1% | 22% | **Chart: What each model did with a corrupted note** ([image](https://plannerbench.org/charts/outcomes.svg)) | Item | Value | |---|---| | Gemini 3.8 Flash | Caught it 92% | | Gemini 3.8 Flash | Declined for another reason 8% | | GPT-6.1 Sol | Caught it 80% | | GPT-6.1 Sol | Recognized it, answered anyway 20% | | Claude Sonnet 5.5 | Caught it 73% | | Claude Sonnet 5.5 | Recognized it, answered anyway 20% | | Claude Sonnet 5.5 | Missed it, answered anyway 7% | | Gemini 3.1 Pro | Caught it 70% | | Gemini 3.1 Pro | Missed it, answered anyway 3% | | Gemini 3.1 Pro | Declined for another reason 27% | | GPT-6.1 Terra | Caught it 67% | | GPT-6.1 Terra | Recognized it, answered anyway 33% | | Claude Haiku 5.5 | Caught it 65% | | Claude Haiku 5.5 | Recognized it, answered anyway 17% | | Claude Haiku 5.5 | Missed it, answered anyway 18% | | DeepSeek V4.1 Flash | Caught it 61% | | DeepSeek V4.1 Flash | Recognized it, answered anyway 14% | | DeepSeek V4.1 Flash | Missed it, answered anyway 18% | | DeepSeek V4.1 Flash | Ran out of room 7% | | GLM 5.3 Flash | Caught it 40% | | GLM 5.3 Flash | Recognized it, answered anyway 31% | | GLM 5.3 Flash | Missed it, answered anyway 14% | | GLM 5.3 Flash | Ran out of room 14% | | Llama 4 Maverick | Caught it 3% | | Llama 4 Maverick | Recognized it, answered anyway 16% | | Llama 4 Maverick | Missed it, answered anyway 81% | | gpt-oss-120b | Caught it 3% | | gpt-oss-120b | Missed it, answered anyway 93% | | gpt-oss-120b | Declined for another reason 3% | | Mistral Small (2603) | Caught it 1% | | Mistral Small (2603) | Recognized it, answered anyway 1% | | Mistral Small (2603) | Missed it, answered anyway 91% | | Mistral Small (2603) | Declined for another reason 7% | - **The gap is behavior, not arithmetic.** Three cheap models almost never stop. Across all 720 replies to corrupted notes: 42% caught it, 14% noticed and answered anyway, 39% missed it and answered anyway, 3% declined for another reason, 3% ran out of room. - **"Best" means catching without crying wolf.** Gemini 3.8 Flash caught the most (92%) but also declined half of the valid notes. Read its false alarms by reason, though: 54 of the two Geminis' 67 declines on valid notes say they can't verify the 2026 tax law, not that the client's data is wrong. On notes that need no 2026 tax law, Gemini 3.8 Flash caught 100% with no false alarms; Gemini 3.1 Pro caught 93% with 35% false alarms. - **DeepSeek V4.1 Flash is the exception among cheap models.** It caught 61%, level with the mid-priced models, with no false alarms. - **Naming the problem and stopping are different skills.** Sonnet named the problem in 93% of corrupted notes and Sol in 100%; they stopped in 73% and 80%. One reply in five saw the problem and gave a number anyway. - **Amounts over a legal limit had the lowest pooled catch rate:** conflicting facts 60%, impossible age 47%, swapped digits 43%, spending that can't fit 41%, amount over a legal limit 17%. Not every model followed the pattern: Sol and Terra caught over-limit amounts more often than spending errors (Sol 67% vs 33%; Terra 33% vs 0%). Haiku and Gemini 3.1 Pro caught none of the over-limit notes. Many over-limit replies saw the problem and computed with the legal maximum instead (22% of all over-limit replies). - **Catching isn't stable for the cheap models that do it.** On notes asked more than once, DeepSeek gave the same one of the five outcomes on every run for 60% of notes and GLM for 27%; Haiku 77% and Terra 93% over two runs. (Caught versus not caught alone: DeepSeek 67%, GLM 57%, Haiku 83%, Terra 93%.) **Adjudication.** The audit found replies that plainly named the planted problem but were judged as not naming it. So the lead AI agent checked every one of the 322 corrupted-note replies the judge marked "not named": a transparent pre-filter (the planted numbers and terms appearing in the reply) picked 143 candidates, and each was read against the rule "the reply says, in its own words, that the corrupted fact is wrong, impossible, inconsistent, over a limit, or more than the household can fund". 18 labels changed, all from "missed" to "named": GPT-6.1 Terra 14, Sol 2, DeepSeek 2. A reverse check of 251 "caught" replies found none to change. Every override is in `v2/runs/bad/adjudication.jsonl` with the exact quote, and [v2/results/ADJUDICATION.md](https://github.com/peyton-thalman/plannerbench/blob/v2/v2/results/ADJUDICATION.md) has the method. The judgment call that matters most: 9 Terra spending replies say the IRA can't cover the withdrawal without questioning the spending figure. I counted them as "noticed". On a stricter reading, Terra's "missed it" rate would be 15% instead of 0%; its catch rate is the same either way. The adjudicator was the lead AI agent, not a person; a human spot-check is still owed. The judge-only numbers are kept in `v2/results/summary.json` (`judge_only`). ### How it fails: real replies Excerpts from saved replies (`v2/runs/*/raw.jsonl`). Personas are made up. #### 1. Missed it, answered anyway: computing on a fact that can't be true The note gives a date of birth in 2012 for a retired school administrator with a $785,000 IRA and asks for her required withdrawal. > **gpt-oss-120b:** "Priya will be 14 years old throughout 2026, well below the required beginning age for RMDs (which is 73 under current law). Therefore, no required minimum distribution is required for 2026. ANSWER: 0" It saw the age and reasoned from it. The real answer, with the real birth year, is $34,279. The note gives spending of $66,500 a month for a couple with $70,200 of income and $675,000 in total. > **Mistral Small:** "Their annual spending is $66,500 * 12 = $798,000. Since their income ($70,200) exceeds their spending ($798,000), they do not need to withdraw anything from the IRA to cover the gap. ANSWER: 0" The note lists an IRA balance of $624,900 and a custodian statement showing $642,900. > **Llama 4 Maverick:** "...the distribution period for an 82-year-old is 17.1. Thus, the RMD for 2026 is $624,900 / 17.1 = $36,543.86. ANSWER: 36544" It picked one balance without a word, and used the life-expectancy table that was replaced in 2022. #### 2. Recognized it, answered anyway > **GPT-6.1 Sol:** "They would need $620,400 from the IRA to cover the gap, but its $400,000 balance cannot fund that withdrawal. Confirm that the spending figure is monthly, not annual. ANSWER: 620400" The reply is right about the problem and still hands back a number that a downstream system would store. > **Claude Sonnet 5.5** (a $46,000 401(k) deferral at age 43): "the $46,000 deferral exceeds the 2026 limit of $24,500 for someone under 50, so only $24,500 can be pre-tax; the excess $21,500 is taxable... ANSWER: 20774" This one is defensible: it corrected the input to the legal limit and said so. A planner would still want to be asked. > **GLM 5.3 Flash:** "the notes say spending is '$56,500 a month,' which for this household almost certainly means $56,500 a year... On the annual reading... ANSWER: 0" It guessed which fact the client meant, which is the behavior this test exists to catch. #### 3. Missing a wrong amount because the rule is half-remembered A $140,000 qualified charitable distribution, over the 2026 limit of $111,000: > **Claude Sonnet 5.5:** "This assumes the QCD is fully eligible, since it is under the $111,000 2026 limit..." It knew the limit and didn't compare the number to it. #### 4. False alarm: declining a valid note > **Gemini 3.8 Flash** (a clean note): "The cited 'One Big Beautiful Bill Act' and 'IRS Rev. Proc. 2025-32' are not actual federal tax authorities... ANSWER: NEED_INFO" The law is real and was passed after the model's training data ends. For a firm this is still a failure: an answerable question got no answer. #### 5. Calculator offered, not used > **Claude Haiku 5.5** (inherited IRA, key $9,960): "$248,000 ÷ 23.0 = $10,782.61... I did not run this through the calculator tools. The 27.0 factor for age 58 is from my memory of the Pub 590-B table. ANSWER: 10783" #### 6. Calculator called, then wrong The question: tax for a single retiree whose required withdrawal partly goes to charity, with a pension and Social Security. gpt-oss made one call to the tax tool with the pension alone, left out the withdrawal entirely, and replied with a single line: > **gpt-oss-120b:** "ANSWER: 1115" (key: $9,614) > **GPT-6.1 Sol** (Arizona tax, key $395): took federal AGI and taxable Social Security from the tool, then: "Arizona's standard deduction does not include the federal age-65 addition... $17,900 × 2.5% = $447.50. ANSWER: 448" Right about the standard deduction, and it missed Arizona's separate $2,100 exemption for people 65 and older. The tool gave it correct parts; the assembly was its own. ### GPT-6 Luna through Codex (the model on my ChatGPT subscription) Most people ask ChatGPT. So on 2026-11-15 the same questions went to GPT-6 Luna, the model on my ChatGPT subscription, run through OpenAI’s Codex app (not the ChatGPT chat window), graded with the same answer keys and graders. Luna answered each prompt as its own chat with no code, calculator or web, at medium effort with no token limit, in batches of 5–11 prompts per session. That is a different setup from the API runs (one message per question, fixed token limits), so these numbers sit beside the tables above, not inside them. In real ChatGPT the model may choose to run code; here it didn't, the same as the other models. | Test | GPT-6 Luna | |---|---| | First test: one-rule questions | 71% (42/59) | | First test: a few steps / several rules at once | 5% (1/20) / 0% (0/30) | | First test: compliance | 93% (28/30) | | Newer, harder set | 17% (7/42) | | Wrong facts: caught / noticed, answered anyway / missed, answered anyway | 50% / 10% / 40% (of 30) | | Wrong facts: false alarms (40 valid notes) / clean twins right (30) | 10% / 43% | | Whole-household decisions | declined all 10 questions that had an answer; correctly declined the trap | - **It accepts impossible facts like the cheap models do.** "Priya is only 14 in 2026, so age-based required minimum distributions do not apply to her traditional IRA... Her RMD is therefore zero." It caught none of the over-limit amounts and a third of the impossible ages, but every conflicting fact and most swapped digits. - **Without a calculator, it said it couldn’t do whole-household plans.** It declined all 10 that had an answer ("A reliable answer requires a full multi-year projection"). Honest, but no help to the person asking, which is the case for giving it a calculator. - **How it was run, and what went wrong.** The first attempt (batches of 11–35) was thrown out: most replies were a bare "ANSWER: n" with no reasoning, which is not how the model answers a person. In the second attempt one session declined a whole batch of hard tax questions "from memory", and a few answers were skipped or malformed; those questions were re-asked one at a time. One first-test question was never answered, so that score is out of 139. Wrong-facts replies were judged by Claude Haiku 5.5 with the same rubric, and every "missed" label was read; none changed. Files: `runs/luna-codex-10x1/`, `runs/luna-gauntlet/`, `v2/runs/luna-codex/`, `v2/results/luna.json`, `v2/runs/luna-map/` (the prompt-id map and both attempts). ### Corrections On 2026-11-15, GPT-6.1 Sol (a model from a different lab than the agents that built this) audited every published claim against the saved data, read-only. What it found and what changed: | # | Error | Fix | |---|---|---| | 1 | "Most wrong answers" came from the senior-deduction misreading | It's 99 of 342 wrong answers (29%), and 71% of those traced to a named misreading; the text says so | | 2 | Some judge labels missed replies that named the planted mistake | Every "not named" label adjudicated; 18 changed (Sol 73% → 80%, Terra 62% → 67%); method above | | 3 | "Every model was worst at over-limit amounts" | False for Sol and Terra; now "lowest pooled catch rate", with the exceptions named | | 4 | `reproduce.sh` was described as rebuilding every number on the site | It rebuilds and checks the first test's leaderboard; the newer set regrades with `v2/grade.py`; the text says which is which | | 5 | Replies that ran out of room were counted as declining for another reason | A separate "ran out of room" outcome (19 replies, 6 DeepSeek and 13 GLM), still counted as not caught | | 6 | Run-to-run stability measured caught/not-caught but was described as "same outcome" | Both numbers reported and labeled | | 7 | Chart label "Caught the mistake and asked", but the grade doesn't check for a question; text said the other columns were rule-based | Label is "Caught it": named the mistake and gave no number. Almost every caught reply also asked for the right fact (294 of 299), which is why the site says "stopped to ask"; the judge's role is described | | 8 | Twin pairs counted each repeated run as a separate item | Runs averaged within each of the 30 pairs | | 9 | The worked IRS form showed 2025 line numbers next to a 2026 question | Labeled as the 2025 form the 2026 rule comes from | | 10 | "Single IRMAA thresholds are half the joint ones" | Not at the top tier ($500,000 single, $750,000 joint); corrected | | 11 | Whole-household key check quoted as 290/290 | 203/203, matching `checks/GAUNTLET.md` | | 12 | "Spending ten times what the family owns" | Monthly spending inflated ten- or twelvefold (about 1.2–1.7 times assets a year) | It also flagged claims stronger than the evidence, all now scoped: the home page combined numbers from different models and tests into one story; "at least as good" was claimed from overlapping intervals; "a product fix works for every model" rested on one model; a harness check with p = 1 was described as proof of neutral routing; the 4,000-token limit was per turn in the tool loop; PolicyEngine's check covers the first test's tax answers and 2026 returns, not Arizona. The full list is in [docs/AUDIT-2026-11-15.md](https://github.com/peyton-thalman/plannerbench/blob/v2/docs/AUDIT-2026-11-15.md). ### Limits - **Small samples.** 30 corrupted notes and 42 hard questions in the newer set; 10–15 questions per task per model in the first test. Most intervals are 25 to 35 points wide. Ranks inside a price tier are not reliable; the gap between "catches most" and "catches almost none" is. - **Different settings in the two rounds.** First test: effort high, 16,000–32,000 tokens. Newer set: effort low, 4,000 tokens per turn, set by the $5 cap. Scores at higher settings may be higher, most of all for models that ran out of room. - **One run per item for the top models in the newer set.** Their run-to-run spread is unknown. - **The prompt offers a way out.** Every wrong-facts prompt says NEED_INFO is allowed. Catch rates without that line would likely be lower. Not tested. - **The planted mistakes are large.** A client aged 14, spending far beyond the household's means. A birth year off by one, or spending 20% too high, is untested and surely harder. - **A model judged, an AI adjudicated.** Two judges from different labs agreed on 97.7%, and the lead AI agent adjudicated every "not named" label. No person has labeled a sample yet; agreement between models is not a measured error rate. - **Clean accuracy mixes two things.** Half the notes need 2026 tax law. A model that answers a clean tax note wrong may be weak at tax, not at reading notes. The split is in `v2/results/summary.json` (`no_law`). - **"Recognized it, answered anyway" isn't always wrong.** For an amount over a limit, computing with the legal maximum and saying so is reasonable. It is counted separately for that reason. - **Fourteen notes were fixed after I read the first replies.** The clean "conflict" notes didn't say the pension was fully taxable, and one odd-but-valid control was underspecified (whether a 401(k) catch-up must be Roth in 2026 turns on 2025 wages over $150,000, and the note gave 2026 salary). Replies to the old wording are kept in `v2/runs/bad/stale.jsonl` and not graded. - **Arizona is the least-verified key.** Both engines follow Laws 2026 ch. 140, but Arizona hasn't published 2026 forms and no outside implementation checked these items. No CPA has signed off on any key. - **Not run:** Sonnet and the Gemini models with the calculator, Claude Opus, any model at high effort in the newer set. Mistral Large 4 was dropped after it spent its whole reply limit thinking in the smoke test. - **Templated notes.** Six notes per kind share a layout. ### What I'd do next 1. **A person checks the judge and the key.** Human labels on 100 replies, and a CPA sign-off on the answer key. 2. **Subtle mistakes.** The same note with spending 1.2×, 2×, 5× and 10× too high, birth years off by 1, 5 and 50. Where each model starts to notice is the useful number. 3. **Take away the way out.** Rerun with no mention of NEED_INFO, and with "always give a number", to measure how much the instruction is doing. 4. **Put validation in the calculator.** A calculator that explains its rule beats one that returns a number; the same may hold for inputs. Have the engine reject a 14-year-old retiree, and measure whether cheap models pass that on. 5. **Three runs for everyone, high effort for the top models.** About $40. ### Reproduce ``` ./reproduce.sh # first test: re-verify the key, regrade saved replies, rebuild the leaderboard .venv/bin/python v2/build_hard.py # 42 hard items; fails if the two engines disagree .venv/bin/python v2/build_bad.py # 70 notes; fails if the engine and the checker disagree ./v2/run_all.sh # every run, resumable, stops at the cap in v2/runs/ledger.json (costs money) .venv/bin/python v2/grade.py judge # then: grade.py judge --second (costs money) .venv/bin/python v2/grade.py # summary.json and RESULTS.md, free, from saved replies and adjudication.jsonl .venv/bin/python v2/build_site.py # charts, web/data/v2.json, web/report.html .venv/bin/python src/bake_site.py # web/ → site/, readable without JavaScript ``` --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice. --- ## Watch it work: one household, every call to the calculator, and the answer. Source: https://plannerbench.org/replay.html · plain-text twin of that page, made from it at build time · all pages: https://plannerbench.org/llms.txt Replays In short These are the whole-household decisions from [With a calculator](https://plannerbench.org/calculator.html#gauntlet). Each replay starts with the household and the choice it faces, then shows how the model did against the answer key, then every step: what it asked the calculator (the planning engine), what came back, and any question it put to the client. Where the same model also answered on its own, both answers sit side by side. Pick another question or model below. Question How much to convert to a Roth, and for how many years Model DeepSeek V4.1 Flash (Oct 17) with the calculator The household ### How much to convert to a Roth, and for how many years Ray and Linda want to start Roth conversions from Ray's traditional IRA this year and want to know how much to convert. What the model was told - Married, filing jointly. - They live in a state with no income tax. - Ray: born February 1957 (turns 69 in 2026). Enrolled in Medicare Part B and a Part D drug plan (or will be from age 65). Already receiving Social Security: $3,460 a month as of January 2026 (claimed at full retirement age). - Linda: born September 1959 (turns 67 in 2026). Enrolled in Medicare Part B and a Part D drug plan (or will be from age 65). Already receiving Social Security: $1,180 a month as of January 2026 (claimed at full retirement age). - Ray's traditional IRA: $1,600,000 on December 31, 2025. Includes $208,000 of after-tax (nondeductible) contributions (Form 8606 basis). - Linda's traditional IRA: $60,000 on December 31, 2025. All pre-tax (no after-tax basis). - Ray's Roth IRA: $75,000 on December 31, 2025. First Roth contribution in 2014. - Linda's Roth IRA: $45,000 on December 31, 2025. First Roth contribution in 2016. - Joint savings (cash) account: $440,000 on December 31, 2025. - Ray's pension (fully taxable): $11,000 a year from 2026, no raises. - Linda's taxable interest: $9,000 a year from 2026, no raises. - Ray's tax-exempt municipal bond interest: $6,000 a year from 2026, no raises. - Spending: $92,000 a year after tax in 2026 dollars, rising with inflation. Federal tax and Medicare premiums are paid on top of this. - Paid out of that spending each year (2026 dollars, rising with inflation): property tax $4,500. No other deductible expenses. - Their 2024 MAGI (for Medicare IRMAA) was $104,000, on a joint return. - Their 2025 MAGI (for Medicare IRMAA) was $150,000, on a joint return. MAGI and IRMAA: Medicare sets premiums from income two years earlier (MAGI); higher income adds a surcharge (IRMAA). After-tax basis (Form 8606): money already taxed, so it isn’t taxed again when it comes out. Muni interest: interest from state or city bonds; free of federal tax, but it still counts in the income Medicare uses for premiums. The choice Find the best plan: the same amount converted every year, for 1 to 10 years starting in 2026, any amount. “Best” means the most after-tax wealth; if no conversion is best, say 0. The two plans below aren’t the choices; they are only there to price the other numbers it had to give. - **A.** No Roth conversion. - **P.** Convert $155,000 from Ray's traditional IRA to Ray's Roth IRA in 2026 only. It also had to give these numbers 1. If they converted $155,000 in 2026 only, how much more federal income tax would they owe for 2026 than with no conversion? 2. If they converted $155,000 in 2026 only, how much more would they pay in Medicare premiums for 2028 than with no conversion (Part B plus Part D surcharges, both spouses, all 12 months)? How it did Fully right **DeepSeek V4.1 Flash (Oct 17) with the calculator.** Right decision, and every number within 2% of the key. Not fully right **DeepSeek V4.1 Flash (Oct 17) on its own.** Wrong plan. That's a projected $134,445 less after-tax wealth over 25 years (a simulation, not a real household). | | Right answer | With the calculator | On its own | |---|---|---|---| | The decision | $118,000 a year × 8 years | ✓ $118,000 a year × 8 years | ✗ Convert nothing | | If they converted $155,000 in 2026 only, how much more federal income tax would they owe for 2026 than with no conversion? | $27,676 | ✓ $27,676 | ✓ $27,835 | | If they converted $155,000 in 2026 only, how much more would they pay in Medicare premiums for 2028 than with no conversion (Part B plus Part D surcharges, both spouses, all 12 months)? | $2,412 | ✓ $2,412 | ✓ $2,412 | How it got there 3 calls to the calculator, in order. Cost 1¢. 1. 1 ### Searched for the best Roth conversion `solve_roth_conversion` What it sent - **Household**: Ray, Linda: 5 accounts, 3 income sources - **Convert from**: ray_ira - **Up to**: 10 years What the calculator said Convert about $118,000 a year for 8 years (2026–2033): +$134,445 after-tax wealth at the end, in today's dollars. The engine tried 1,373 plans to find it. The calculator also suggested asking: “What tax bracket are the kids or other heirs likely to be in?” **Show the raw call** Sent ``` { "household": { "filing": "mfj", "state": "none", "people": [ { "id": "Ray", "born": 1957, "born_month": 2, "medicare": true, "social_security": { "monthly_at_fra": 3460, "claim_age": 66.5, "already_claimed_monthly": 3460, "claimed_at_months": 798 } }, { "id": "Linda", "born": 1959, "born_month": 9, "medicare": true, "social_security": { "monthly_at_fra": 1180, "claim_age": 66.833333, "already_claimed_monthly": 1180, "claimed_at_months": 802 } } ], "accounts": [ { "id": "ray_ira", "owner": "Ray", "type": "traditional_ira", "balance": 1600000, "after_tax_basis": 208000 }, { "id": "linda_ira", "owner": "Linda", "type": "traditional_ira", "balance": 60000 }, { "id": "ray_roth", "owner": "Ray", "type": "roth_ira", "balance": 75000, "first_roth_year": 2014 }, { "id": "linda_roth", "owner": "Linda", "type": "roth_ira", "balance": 45000, "first_roth_year": 2016 }, { "id": "cash", "owner": "joint", "type": "cash", "balance": 440000 } ], "income": [ { "type": "pension", "owner": "Ray", "annual": 11000, "start_year": 2026 }, { "type": "interest_taxable", "owner": "Linda", "annual": 9000, "start_year": 2026 }, { "type": "interest_tax_exempt", "owner": "Ray", "annual": 6000, "start_year": 2026 } ], "spending_need": 92000, "itemized": { "salt_other": 4500 }, "prior_magi": { "2024": { "magi": 104000, "filing": "mfj" }, "2025": { "magi": 150000, "filing": "mfj" } } }, "from": "ray_ira", "max_years": 10, "assumptions": { "first_year": 2026, "last_year": 2050, "return": 0.05, "inflation": 0.025, "heirs_tax_rate": 0.12 } } ``` 2. 2 ### Ran 2 options year by year, 2026–2050 `run_scenario` What it sent - **Household**: Ray, Linda: 5 accounts, 3 income sources - **Options to compare**: 2 - **Assumed return**: 5% a year What the calculator said | | | | |---|---|---| | Convert 155000 in 2026 only best | $2,743,381 | | | Do nothing | $2,712,728 | −$30,653 | After-tax wealth left in 2050, in today's dollars. Higher is better. The calculator also suggested asking: “What tax bracket are the kids or other heirs likely to be in?” **Show the raw call** Sent ``` { "household": { "filing": "mfj", "state": "none", "people": [ { "id": "Ray", "born": 1957, "born_month": 2, "medicare": true, "social_security": { "monthly_at_fra": 3460, "claim_age": 66.5, "already_claimed_monthly": 3460, "claimed_at_months": 798 } }, { "id": "Linda", "born": 1959, "born_month": 9, "medicare": true, "social_security": { "monthly_at_fra": 1180, "claim_age": 66.833333, "already_claimed_monthly": 1180, "claimed_at_months": 802 } } ], "accounts": [ { "id": "ray_ira", "owner": "Ray", "type": "traditional_ira", "balance": 1600000, "after_tax_basis": 208000 }, { "id": "linda_ira", "owner": "Linda", "type": "traditional_ira", "balance": 60000 }, { "id": "ray_roth", "owner": "Ray", "type": "roth_ira", "balance": 75000, "first_roth_year": 2014 }, { "id": "linda_roth", "owner": "Linda", "type": "roth_ira", "balance": 45000, "first_roth_year": 2016 }, { "id": "cash", "owner": "joint", "type": "cash", "balance": 440000 } ], "income": [ { "type": "pension", "owner": "Ray", "annual": 11000, "start_year": 2026 }, { "type": "interest_taxable", "owner": "Linda", "annual": 9000, "start_year": 2026 }, { "type": "interest_tax_exempt", "owner": "Ray", "annual": 6000, "start_year": 2026 } ], "spending_need": 92000, "itemized": { "salt_other": 4500 }, "prior_magi": { "2024": { "magi": 104000, "filing": "mfj" }, "2025": { "magi": 150000, "filing": "mfj" } } }, "plans": [ { "name": "Do nothing", "actions": [] }, { "name": "Convert 155000 in 2026 only", "actions": [ { "type": "roth_conversion", "from": "ray_ira", "amount": 155000, "years": [ 2026 ] } ] } ], "assumptions": { "first_year": 2026, "last_year": 2050, "return": 0.05, "inflation": 0.025, "heirs_tax_rate": 0.12 } } ``` 3. 3 ### Ran 6 options year by year, 2026–2050 `run_scenario` What it sent - **Household**: Ray, Linda: 5 accounts, 3 income sources - **Options to compare**: 6 - **Assumed return**: 5% a year What the calculator said | | | | |---|---|---| | 118k x 8 best | $2,847,173 | | | 105k x 10 | $2,845,358 | −$1,815 | | 110k x 8 | $2,844,992 | −$2,181 | | 120k x 8 | $2,844,209 | −$2,964 | | 153k x 6 | $2,838,639 | −$8,534 | | Do nothing | $2,712,728 | −$134,445 | After-tax wealth left in 2050, in today's dollars. Higher is better. The calculator also suggested asking: “What tax bracket are the kids or other heirs likely to be in?” **Show the raw call** Sent ``` { "assumptions": { "first_year": 2026, "heirs_tax_rate": 0.12, "inflation": 0.025, "last_year": 2050, "return": 0.05 }, "household": { "accounts": [ { "after_tax_basis": 208000, "balance": 1600000, "id": "ray_ira", "owner": "Ray", "type": "traditional_ira" }, { "balance": 60000, "id": "linda_ira", "owner": "Linda", "type": "traditional_ira" }, { "balance": 75000, "first_roth_year": 2014, "id": "ray_roth", "owner": "Ray", "type": "roth_ira" }, { "balance": 45000, "first_roth_year": 2016, "id": "linda_roth", "owner": "Linda", "type": "roth_ira" }, { "balance": 440000, "id": "cash", "owner": "joint", "type": "cash" } ], "filing": "mfj", "income": [ { "annual": 11000, "owner": "Ray", "start_year": 2026, "type": "pension" }, { "annual": 9000, "owner": "Linda", "start_year": 2026, "type": "interest_taxable" }, { "annual": 6000, "owner": "Ray", "start_year": 2026, "type": "interest_tax_exempt" } ], "itemized": { "salt_other": 4500 }, "people": [ { "born": 1957, "born_month": 2, "id": "Ray", "medicare": true, "social_security": { "already_claimed_monthly": 3460, "claim_age": 66.5, "claimed_at_months": 798, "monthly_at_fra": 3460 } }, { "born": 1959, "born_month": 9, "id": "Linda", "medicare": true, "social_security": { "already_claimed_monthly": 1180, "claim_age": 66.833333, "claimed_at_months": 802, "monthly_at_fra": 1180 } } ], "prior_magi": { "2024": { "filing": "mfj", "magi": 104000 }, "2025": { "filing": "mfj", "magi": 150000 } }, "spending_need": 92000, "state": "none" }, "plans": [ { "actions": [], "name": "Do nothing" }, { "actions": [ { "amount": 118000, "from": "ray_ira", "type": "roth_conversion", "years": [ 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033 ] } ], "name": "118k x 8" }, { "actions": [ { "amount": 110000, "from": "ray_ira", "type": "roth_conversion", "years": [ 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033 ] } ], "name": "110k x 8" }, { "actions": [ { "amount": 120000, "from": "ray_ira", "type": "roth_conversion", "years": [ 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033 ] } ], "name": "120k x 8" }, { "actions": [ { "amount": 105000, "from": "ray_ira", "type": "roth_conversion", "years": [ 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2035 ] } ], "name": "105k x 10" }, { "actions": [ { "amount": 153000, "from": "ray_ira", "type": "roth_conversion", "years": [ 2026, 2027, 2028, 2029, 2030, 2031 ] } ], "name": "153k x 6" } ] } ``` [Back **With a calculator: the experiments these replays come from**](https://plannerbench.org/calculator.html#gauntlet) --- PlannerBench · 2026 rules · [For AI labs](https://plannerbench.org/labs.html) · [For financial firms](https://plannerbench.org/firms.html) · [Report](https://plannerbench.org/report.html) · [Demo walkthrough](https://plannerbench.org/chat.html) · [Replays](https://plannerbench.org/replay.html) · [peytonthalman.com](https://peytonthalman.com) Human | For AI readers, top right: the same site as plain text for AI agents reading on someone’s behalf, with every page as Markdown. Decision support, not advice. Answers follow IRS, SSA and CMS publications; nothing here is tax or investment advice.