When The AI Gives the Correct Answer But Completely Misses the Point

Late at night, a foreign graduate student preparing to move to the UK typed a question into an AI-powered government assistant.

She had just noticed a small discrepancy between the employment dates listed in her visa paperwork and the dates appearing on a supporting document from a previous internship.

Is this likely to create a problem with my application?”

The system responded efficiently. It explained the official requirements, linked to the relevant immigration guidance, and clarified which documents needed to be submitted.

Technically, the answer was correct.

But she left the interaction still uncertain.

Because what she was really asking was not:

“What are the rules?”

She was asking:

“Should I be worried?”
 “How much risk is there here?”
 “What would someone experienced advise me to do next?”

That distinction is becoming increasingly important as organizations deploy AI systems at scale.

In recent pilots of Government Digital Service’s GOV.UK Chat, more than 10,000 users asked about 26,000 questions about taxes, visas, childcare, benefits, and other public services. By most technical measures, the pilots were successful: user satisfaction was relatively high, and the system’s accuracy improved from 76% to 90% during testing.

But one of the most interesting findings had less to do with accuracy than ambiguity.

Researchers observed that many users were not simply looking for information. They were trying to understand how rules applied to their specific situation. In response, the GOV.UK team introduced more clarifying questions because users often phrased requests in ways the system could not confidently interpret.

This captures one of the biggest misunderstandings organizations currently have about AI:

AI often fails while being right, and it fails because we don’t know how to think with it, or because it is not designed to think with us.

That is the central thesis of Smarter Together: How We Think, Feel and Decide with AI (Cambridge University Press), a new book I co-authored with Michael Joffe (Global Marketing at Google and NYU Adjunct Professor) and Michael Clarke (Senior Product Director at Shopify, and NYU Adjunct Professor). The book explores what happens when humans begin making decisions alongside systems that sound intelligent, persuasive, and confident, but do not actually think the way humans do.

And this is becoming obvious in consumer research.

Over the last year, I’ve spoken with research teams experimenting with AI moderation, synthetic consumers, automated summaries, observational AI, and AI-assisted analysis. The excitement is real as many of these tools are genuinely transformative:

• summarizing hundreds of open ends in minutes,

• scaling qualitative work,

• accelerating concept exploration,

• identifying patterns humans might miss,

• and helping researchers move from isolated datapoints to broader behavioral understanding.

Across industries, organizations are racing to integrate AI into products, workflows, customer experiences, and decision-making. New pilots launch almost weekly, investment continues to surge, and the systems themselves are improving at a fast pace. 

However, technical capability alone does not automatically translate into meaningful impact. It turns out that successful AI adoption depends on more than speed or accuracy, but on whether people know how to interpret, question, and make decisions alongside increasingly intelligent systems; and importantly, on whether users trust the AI.

In research environments, this differentiation is remarkable because consumer research has never really been about processing information, but about interpreting human behavior.

The hesitation before an answer.
The contradiction in the middle of an interview.
The emotional tension underneath a rational explanation.

That is usually where the real insight lives.

One of the concepts we explore in Smarter Together is something psychologists call processing fluency: when information is easy to process, humans are more likely to trust it. AI systems are extraordinarily fluent. They communicate clearly, confidently, and coherently. Even riskier than AI hallucinations is that it can sound convincing before humans have fully interrogated the thinking underneath it.

The answer here is to slow down thinking through better human-AI interaction. Sometimes good decision-making requires what we call positive friction: moments that encourage reflection, challenge assumptions, or surface uncertainty instead of hiding it. Some of the best insights in research emerge from the five uncomfortable minutes when nobody in the room agrees on what the data means.

Emotion matters too. Consumers rarely make decisions based purely on information. Linguistically, a response may appear complete while psychologically the real decision driver remains unstated. Perceived empathy, the feeling that someone or something genuinely understands your situation, significantly increases engagement, openness, and depth of response. When people feel understood, they reveal more, reflect more deeply, and provide richer insights. That is one of the reasons emotionally intelligent interaction matters so much in both research and AI design.

And none of this happens in a cultural vacuum. AI systems are often designed as if communication were universal. But trust, uncertainty, directness, and explanation are interpreted very differently across cultures. The same AI response can feel transparent in one context and deeply unsettling in another.

At SXSW earlier this year, we spent a great deal of time discussing this: organizations are increasingly optimizing AI systems for efficiency when they should also be designing them around human cognition.

We are already starting to see early versions of this shift emerge across the industry: AI moderators designed to probe tension instead of simply extracting answers, synthetic personas grounded in behavioral science rather than demographics alone, and observational AI systems capable of identifying subtle behavioral patterns from real-world consumer interactions.

These systems point toward a much larger shift: from replacing human judgment to augmenting it. That shift sits at the center of Smarter Together.

And it also represents the biggest opportunity for the insights industry.

The organizations that create the most value from AI will be the ones that best understand how humans think, trust, and decide alongside them.