Your AI sounds confident but that doesn't mean it's right.

Your AI sounds confident but that doesn't mean it's right.
25 06 2026

You've probably experienced this before when you ask an AI a question, it replies in a clear, confident, structured manner, only for you to realise later that it was wrong. No hesitation, no nuance, just a mistake, delivered as if it were perfectly sure of itself. If you prompt further, it will tell you that it hallucinated.

This isn’t a coincidence, and it’s not only happening to you. It's a direct consequence of the way AI is built today. What is unsettling is that the AI gives you no signal when it crosses the line from fact to fiction. The tone remains calm, the writing style stays polished, the thanks are engaging, and the result is the same, whether it's right or making things up.

Why this should matter to you
 

You may not work in law or medicine. But you probably use AI to draft emails, do research before making a purchase, help your kids with their homework, or familiarise yourself with a subject you're not an expert in. Every time you do, you assume the answer is reliable.

The hard part is that unreliable AI doesn't seem unreliable. There are no warnings, no disclaimers, and no change in tone when it switches from a real fact to an invented one. The confidence is constant. That's exactly what makes it so easy to miss, and so hard to spot on your own.

Why does it happen?
 

Well, the reason is simpler than you might think. Today's AI is trained to mimic humans and to please. It isn't trained to tell the truth but trained to sound right. When asked a question, it does not look for the answer, but predicts, word for word, what a convincing human response would be.

Most of the time, the most convincing answer happens to be true. So, it works. But when the AI doesn’t know, it doesn’t stay silent or say “I’m not sure”; it produces something that sounds just as convincing, because this is the only thing it was designed to do. It has no way of distinguishing its real answers from the ones it makes up, because no one has ever taught your AI the difference.

This also explains something surprising, newer, more powerful models don't automatically become more trustworthy. Getting better at sounding human means getting better at sounding right, including when they make mistakes.

It's not just a chatbot quirk
 

This shows up in the serious tools people rely on, not just casual chats. Researchers at Stanford tested the AI research tools that lawyers use every day, the ones sold by LexisNexis and Thomson Reuters and marketed as nearly error-free. Those tools gave incorrect or unsupported answers between 17% and 33% of the time. The study was peer-reviewed and published in a leading legal journal.

In medicine, a team at Mount Sinai tried a revealing experiment. They slipped one fake detail, a made-up condition or nonexistent lab test, into otherwise normal patient descriptions. They then asked leading AI models for their take. Instead of catching the error, the models ran with it about two thirds of the time, confidently building on something that was never true. The work was published in Communications Medicine, a peer-reviewed publication rom the Nature Portfolio journals.

And in everyday shopping, researchers at UC San Diego looked at AI tools that summarize customer reviews. In more than a quarter of cases, the summaries quietly shifted the overall tone, making products sound better or worse than the reviews said. In a test with people, shoppers' behavior changed, with 32% more likely to want to buy a product after reading the AI's summary than after reading the actual reviews.

These aren't trick questions designed to break the AI. They're ordinary tasks, and the same pattern shows up every time.

Patching the problem isn't fixing it 

 

The usual industry approach is to build the most capable system possible, after which companies add safety measures, fact checking, confidence scores, human review and filters. These actions help, but they don't change the models’ fundamental behaviour. This reactivity stems from a deeper issue: we are still treating AI as a black box. 

Underneath it all, the AI is designed to sound convincing, and because we don't fully understand how it processes information, our fixes remain mere patches. A growing number of researchers argue that simply altering training methods isn't enough. Trustworthiness can't be an afterthought; it has to be built in from a foundation of clear understanding.
 

A different way to build it
 

That's the idea behind the work at LawZero: safe-by-design. Instead of building a system that's clever first and trustworthy second, what if trustworthiness was baked into the design, from the beginning? 

The Scientist AI, developed at LawZero, is built around two simple ideas that inject trustworthiness and honesty in the model.

- It weighs what it reads
A vast majority of AIs treat almost every online content it sees as equally true, whether it comes from a careful study or a random post online. The Scientist AI tracks where a claim came from and weighs the evidence, instead of just repeating a statement or idea it has seen most often.

- It isn't trying to please you
Most AIs learn from whether people like their answers, slowly learning to tell you what you want to hear. The Scientist AI gets no such reward. Its only job is to be accurate, to understand the world, not to win you over. The goal is a system that's genuinely intelligent but has no reason to spin, flatter, or pursue a hidden agenda.

None of this is just theory as LawZero is already publishing research and developing the model. The point is simple: AI can be both capable and trustworthy, and it's worth getting that right before we all quietly come to rely on AI that isn't.
 

Sources: