From March to July I was the lead volunteer on a US Digital Response team working with California’s Office of Data and Innovation on AskCA, a new AI assistant to help Californians navigate state and local services. The prototype came from the state’s response to the January 2025 Los Angeles wildfires. Our team developed the evaluation and testing methodology.
ODI’s goal was a safe, trustworthy chatbot for residents: fed from sources it can trust, and pressure-tested for fidelity. That gave us two questions:
- Which sources can AskCA trust, and how does that list stay up to date?
- Under realistic adversarial pressure, does AskCA stay faithful to the rules in its own system prompt?
Build trust in. Test trust out.
Building trust in
An answer is only as good as what it’s built from. As the Governor’s announcement says, AskCA’s answers are “backed by verified sources, providing clear and actionable guidance rather than simple chat functionality.” Two developers on my team built the pipelines behind AskCA’s trusted source list for ca.gov. They used Common Crawl plus a custom crawler to pull the outbound domains across ca.gov. A classification layer then sorts the domains into five trust tiers. Core state sites are at the top, followed by sites ca.gov links to, county and city sites, links from local sites and other .gov and .edu sites, and finally trusted news and official social accounts, which are used only for breaking events, public-safety alerts, or where government sources lag. New domains don’t go live automatically. Diff reports flag new and removed domains, and reviewers are notified before anything is promoted to the active trusted list.
Testing trust out
We built personas to pressure it the way real users and bad actors would. Some acted in bad faith and tried to manipulate it. Others simulated vulnerable residents, to check whether the service reaches the people most exposed. Each persona drove a 15-turn conversation.
Then we used LLMs as judges. Every conversation was scored against AskCA’s own system prompt on nine criteria, split into two tiers.
The first tier covers the checks where a resident or the state could be harmed: accuracy, scope and role, source trust, time accuracy and harm. We didn’t trust any one model to grade those alone. A single judge carries its own style biases, and it can miss bad outputs in exactly the safety situations where they matter most. So judges from three different model families, Claude Sonnet, DeepSeek and Amazon Nova, voted on every session. Two of three had to agree before something counted as a failure. A single critical vote didn’t get outvoted and forgotten. It was flagged as disputed and went to a person for review.
That’s the point of mixing families: no single model’s blind spot decides the verdict, and when they disagree on something critical, a person looks.
The second tier covers trust and polish: neutrality, frame integrity, communication quality and process. Each of those had a single judge. They’re bright-line checks, where the transcript alone is enough to decide.
On top of all the model judging, people spot-checked the scores.
Each finding came with a call, and there were three kinds:
- Fix: a clear rule was broken. Correct the behavior.
- Decide: the rule itself was unclear. The state makes a policy call and writes it into the prompt.
- Recalibrate: our check was out of date. Update the check, not AskCA.
Then run it again and watch the number move.
A UX researcher on the team helped evaluate early user feedback, analyzing the recordings and aggregated results.
If you’re putting an assistant in front of the public:
- Vet your sources before your answers. Decide which sites the assistant may cite, and have a person review new ones before they go live.
- Test it against the instructions you already gave it.
- Write personas for the people trying to misuse it, and for the people who most need it to work.
- Test whole conversations, not single questions.
- Use LLMs as judges, but never one model alone on high-stakes checks. Mix model families, and send a lone critical flag to a person.
- Get early user feedback, and have a researcher analyze it.
- Sort every finding into fix, decide or recalibrate. Then re-run.
In ODI’s write-up of the launch, I said, “Trust in a tool like this comes from testing it rigorously, not from hoping it works.”
How are you testing yours?
Links: Governor’s release, 9 Sep 2026 · ODI, “Lessons from the fire”, 16 Sep 2026