For financial firms

Your clients are already asking AI. Even in a retirement community where most people have an advisor, prospects tell me they ask ChatGPT first. One told me a 529 question felt too dumb to ask me. They trust it because it seems right and Google’s AI agrees.

What I found

  • On single rules, good models are almost always right.
  • On real retirement questions, where taxes, Social Security, and Medicare interact, it depends heavily on the model. The cheap and free models people actually use scored under 30%.
  • When a client gets their own facts wrong, most models give confident advice anyway.

What fixes it, without paying for the most expensive model

  • Ask the questions that matter in a set order, so nothing gets assumed.
  • Give the AI a calculator that shows the rule behind every number.
  • Have it flag facts that don’t make sense before it answers.

What a firm could do with this

  • Know which questions need an advisor. The test data shows where AI does well and where it breaks.
  • Offer a “check your AI answer” tool on your website. Someone brings the question they asked ChatGPT. The tool collects the facts that matter, flags what needs a professional, and asks if they’d like an advisor to reach out. People are using AI either way, so the firm might as well offer the version that checks itself.
  • Keep costs down. Structure and checks let cheap models do the job, which matters across thousands of advisors and their clients.

How I built it → Technical details →