PlannerBench
Can AI answer retirement questions correctly? I tested it.
PlannerBench is a set of retirement planning questions with known right answers. I give the questions to AI models and grade what they say.
The questions cover taxes, Social Security, Medicare, required minimum distributions, and Roth conversions under 2026 rules. I built a planning engine to work out the correct answers, and a second, separately written program to double-check them.
Why I built it
I’m a financial advisor, and I meet a lot of retirees by knocking doors. Many of them tell me they ask ChatGPT their money questions. When I ask how they know the answer is right, they say they Googled it and the top result agreed. That top result is usually Google’s AI. I wanted a real way to check, so I built one.
Results
- Questions with one rule: The top models got about 97% right.
- Questions where several rules interact: This is where most models fall apart. On the 42 hardest questions, GPT-6.1 Sol got 98%, Claude Sonnet 5.5 got 62%, and GPT-6 Luna got 17%.* Cheap and free models were all under 30%.
- With a calculator that shows its work: When the calculator gave the rule and the IRS form line along with the number, Claude Haiku went from 40% to 100%.
- When the client’s information is wrong: I put one impossible fact in each client file, like a birthday that makes the client 14. Most models gave advice anyway. GPT-6 Luna gave retirement advice to a 14-year-old. Even the best models missed 20–27% of these.
If you use AI for retirement questions
Check the ages, balances, and income it used. Ask which rule it’s applying. If it can’t tell you, or it got your facts wrong, don’t act on the answer. For decisions you can’t undo, like when to start Social Security, talk to a person.
For financial firms
Your clients are already asking AI before they ask you. Most can’t tell when it’s wrong, and the AI usually won’t tell them when the facts they gave it are wrong.
For AI labs
Every number answer here is exact and checked, so a script can grade it without a person or another AI deciding what’s right. That makes it useful for testing and training models on the money questions people are already asking.
Where this is going
I believe the best AI models will soon be able to do financial planning at the level of the best human advisors. The free models will catch up to today’s best paid ones. This needs to be done at a bigger scale than I can manage on my own, and I’d like to take it further as part of a team.
See example questions → How I built and checked it → peytonthalman.com
*GPT-6 Luna is the model behind the regular ChatGPT app. I tested it through Codex with code and web access off. That’s close to the ChatGPT app, but not identical.
Decision support, not advice. Answers follow IRS, SSA, and CMS publications. Nothing on this site is tax or investment advice.