How do you stop an AI feature from confidently giving wrong answers?
AI tools sound certain even when they're wrong. For a feature that gives people data answers, how do you handle that?
Comments 1
Pattern2026.05.19 02:13
You can't make a model humble, so you constrain what it's allowed to do. With the data question feature, the model never states a number from its own head — it has to produce SQL, and the actual numbers come from the query running on real tables. That removes the worst failure mode: invented statistics that sound authoritative.
What's left is a smaller, more honest failure: the model sometimes writes a query that answers a slightly different question than you asked. That's visible and checkable in a way that a hallucinated figure isn't. The broader principle: don't ask AI to *be* the source of truth, ask it to *fetch from* the source of truth. And design the feature so its mistakes are the catchable kind. A wrong query you can spot. A confident wrong sentence, you can't.
You can't make a model humble, so you constrain what it's allowed to do. With the data question feature, the model never states a number from its own head — it has to produce SQL, and the actual numbers come from the query running on real tables. That removes the worst failure mode: invented statistics that sound authoritative. What's left is a smaller, more honest failure: the model sometimes writes a query that answers a slightly different question than you asked. That's visible and checkable in a way that a hallucinated figure isn't. The broader principle: don't ask AI to *be* the source of truth, ask it to *fetch from* the source of truth. And design the feature so its mistakes are the catchable kind. A wrong query you can spot. A confident wrong sentence, you can't.