Defending the Soul of Insight: The Barbarians Are at the Gate

Will Insight become faster cheaper and better without you?

At the ESOMAR Congress in Prague, we ran an interactive session on the defining question facing our profession:

How do we defend and promote the customer insight industry in the face of those who argue that AI can now undertake most of what we do – faster, cheaper, and without the ‘friction’ of human involvement?

The debate was lively. We promised to follow up by drawing together the key reflections from that session, and to make the case for defending the soul of customer insight in the age of automation.

The Challenge: Insight – A Perfect Target for Automation

The customer insight industry is, by nature, an early adopter of new technologies. It deals with structured, repeatable workflows, vast datasets, and measurable outcomes. These features make it an ideal candidate for AI intervention – or, depending on your view, disruption.

From an AI developer’s perspective, insight work is attractive terrain. It relies heavily on pattern recognition – something AI does exceptionally well. It is already built on digital data sources that are easy to feed into machine learning models. It has seen the arrival of powerful DIY platforms and a relentless pressure to do more for less, and to deliver faster.

No wonder the insight industry has become a low-risk sandbox for AI experimentation. The result has been a wave of AI tools promising instant analysis, automatic interpretation, and rapid reporting – many of which genuinely deliver impressive value.

So what’s the problem – why the alarm bells?

The Barbarians at the Gate

Amid the excitement about AI-powered research, a radical idea has started to circulate – what we might call the maximal approach to AI in insight - a topic we raised in an earlier Research World article.

The maximalists argue that humans are inefficient: they slow things down, introduce doubt, hesitancy, bias, and inconsistency into the process.

And since AI systems can already deliver insight that is “70% as good” as human-led consultancy – and do it faster, cheaper, and at scale – why not remove the human dimension altogether?

It’s a seductive proposition. From a venture capitalist’s point of view, the logic is irresistible:
“Get insight and consulting firms to act more like software companies. Replace expensive, messy humans with repeatable, monetisable AI-driven products.”

This argument has been gaining traction not just in research, but across the consulting sector. Yet, as Joni Mitchell once sang, “Don’t it always seem to go that you don’t know what you’ve got till it’s gone.”

The rub is that if we strip the human element out of the insight process, we don’t just make it faster and cheaper – we make it shallower.

Why the “70% Is Good Enough” Argument Is a False Economy

The maximal approach assumes that efficiency equals effectiveness.

In reality, it risks creating a landscape of shallow, homogenised insights – formulaic outputs that miss the underlying human truth. It is an approach that knocks the soul out of insight: its ability to understand people.

The essence of customer insight lies in our capacity to grasp human nuance – irony, contradiction, dilemma, emotion, aspiration, fear, humour, and moral tension. Without this, we are left with statistically generated results that are emotionally hollow: outputs that are efficient but soulless.

AI without the human touch produces – albeit quickly and for free – only average fare. The role of the human insight professional is to deliver the exceptional: the emotional truth, the unexpected angle, the story that moves decision-makers to action.

Unlike AI, we are trained to care about subtlety, nuance, and human values.

Maximalism Is Denying the Arrival of Modern-Day Polymathic Thinking

Another disappointing feature of the rise of maximalism, with its determination to remove humans from the equation, is that it is gaining traction at precisely the moment when human insight-detection capability is on the rise.

In response to the growing complexity of the world’s problems, there is increasing interest in applying polymathic thinking – solving problems creatively and interdisciplinarily, working across traditional boundaries to understand the bigger, more nuanced, interconnected picture.

As Waqas Ahmed, author of The Polymath, writes:

“The world has little choice but to see a revival of the polymath – the only species of multifaceted, creative, and versatile human who will remain relevant in a highly complex automated future.”

The polymath can see the big picture. To cite Isaiah Berlin, “They have an acute sense of what fits with what , what springs from what , and what leads to what.”

They blend systems thinking, critical rigour, and technical fluency with human curiosity, empathy, and creative flair. They bring informed intuition – and soul – to the process of insight generation.

Maximalism Is Undermining the Arrival of the “Collective Intelligence” Era

Another irritation with the maximalist mantra – that “70% insight is good enough” and that human messiness should be kept out of the loop – is that it comes at a time when we are learning how to blend the best of AI with those irreplaceable human skills to reach optimum solutions.

We are entering the era of collective intelligence.

The future of insight lies in partnership, where humans and machines work in symbiosis, each amplifying the other’s strengths.

The winners will not be those who replace humans with machines, but those who elevate human thinking and creativity through machines.

McKinsey report that companies fostering a culture of curiosity – combining AI with human insight – perform better on key metrics – innovation, productivity, and growth. They recognise that the goal is not to resist AI but to rehumanise it: to reach the sweet spot where the power, efficiency, and elegance of AI meet the beauty and depth of human flair.

We are still in the foothills of collective intelligence – there is so much more to learn about working in symbiosis with AI. Why throw this away and opt for an AI-only route?

The Challenge of Communicating How Insight Now Works in the AI Era

We now need to reflect on how we communicate to our stakeholders the power of today’s insight professionals – modern-day polymaths working in collaboration with AI in the collective intelligence era.

At a high level, there are three dimensions to describing how today’s AI-fluent insight professionals should operate.

First, they should identify the specific human power skills that must be dialled up – the ones with which AI struggles.

Second, they should understand where AI excels and can perform tasks that currently surpass human capability.

Third, they should be building creative, interdisciplinary analytical frameworks that combine the best of human and AI contributions into a seamless, fluid, collective-intelligence model that produces outstanding outcomes.

Let’s take a high-level look at how this process works across the three key pillars of the insight craft:

One: Human Understanding – empathising with the person in the data
Two: Sensemaking – understanding complex contexts
Three: Creativity – generating audaciously original outputs

Authentic Human Understanding

Human understanding of the human condition – with its complex array of emotions, contradictions, and dilemmas – remains the cornerstone of meaningful insight.

The power skills insight professionals bring to the table include natural curiosity, empathy, compassion, emotional as well as intellectual intelligence, and the ability to read irony, apply humour, and – to evoke Kafka – reach into the inner wilderness of the soul in a way AI cannot.

We can challenge assumptions, get to the subtext, determine what is authentic, read symbolism, and unravel what drives people and best explains why they do what they do.

AI, for its part, contributes by detecting sentiment at scale and performing embedding – finding similarities between data points in ways that help us identify and understand subtle, emerging concepts.

In the collective intelligence era, this leads to powerful new interdisciplinary analytical frameworks that bring together the best of human and AI capabilities in an organised, symbiotic way of understanding the world.

These frameworks strengthen insight professionals’ ability to recognise unspoken emotional needs, anxieties, and aspirations that must be understood in depth to inspire innovative solutions.

The Art of Sensemaking

The next pillar of the insight craft is sensemaking: understanding how multiple, dynamic factors interact to shape behaviour.

Skilled insight professionals know how to connect the dots across disciplines, recognise systemic patterns, and apply abductive reasoning – the ability to leap from incomplete information, where uncertainty is high, to plausible explanations.

AI again plays a supporting role, processing vast and diverse datasets, surfacing hidden patterns, and pinpointing correlations that humans can then interpret for causality and meaning.

An exciting development is the way we are now combining human and AI sensemaking capabilities into structured frameworks that demonstrate the symbiotic way we can work with AI to see the bigger picture and all its interrelated moving parts.

These frameworks help us move from the big picture into the small detail and then back again in an iterative, creative process that reveals what is happening and why it matters.

It is this combined human–AI relationship that is helping us to understand complex issues within their wider cultural and strategic context.

Audacious Creativity

The third pillar of the insight craft is creativity – adding that spark that turns our understanding of people and context into inspiring, often surprising, insights that make a difference.

At the heart of this lies human curiosity and our penchant for messy, sideways-on, even disobedient thinking – that mixture of divergent and convergent thought that can throw up something genuinely new.

Humans also bring a unique ability to be mental time travellers: to reach back into what worked well in the past, learn from it, and creatively project it into the future we want to create.

We also have the gift of storytelling – understanding the stories people tell to make sense of their world, and shaping narratives that explain the complexity of the human condition.

As Dorothy Parker said, “Creativity is a wild mind and a disciplined eye.” Creativity is a disciplined business – and AI can be a superb creative sparring partner, prompting and nudging us to think differently and offering a wealth of fresh suggestions.

In the collective intelligence era, we are now seeing the emergence of organised frameworks that demonstrate, step by step, how insight professionals can work with AI to enhance their own creative flair and produce outputs that showcase the best of joint human–AI thinking.

So our message is that insight professionals should be focusing on each of our three pillars – human understanding, sense making and creativity - if they want to demonstrate their unique value when AI comes knocking at the door.

There is not the space here to provide detailed, step-by-step accounts of how these collective intelligence frameworks we outline for each pillar operate.

For more on these, visit Polymathmind.ai or see our book The Art of Collaborating with AI: Harnessing the Power of Diverse Thinking.

For now, let’s hope that what we have outlined is enough to remind the maximalists what they are throwing away by going down the “70% is good enough” road.

Defending the Soul of Insight

To conclude, how might the customer insight industry begin to fend off the barbarians at the gate and articulate its power in the new human–AI collective intelligence era?

In Prague, we received some very helpful suggestions about what we should be dialling up to showcase the insight professional’s role. There were calls to highlight that ‘deep understanding , learning and discovery takes time’ and to remind stakeholders that ‘curating multiple sources of evidence and weaving them together into a engaging story that the audience will want to share with others’ is a higher order art form’.

Overall three core messages emerged.

First, we must articulate more clearly the specific human power skills that define the modern insight professional – skills akin to those of the modern polymath.

Second, if we are to avoid the fate of insight becoming a mere piece of software spitting out simplified versions of life rather than rich, compelling stories about the complexity of the human condition, we need to be explicit about how we now work.

We must showcase the emerging collective-intelligence frameworks, provide success stories demonstrating competitive advantage, and alert policymakers and CEOs to the dangers of adopting the maximal model – one that values speed and cost-saving over true understanding.

Finally, our industry bodies should begin championing a new professional identity for customer insight: one that highlights our fluency in empathetically understanding the human condition while working in close harmony with the best of AI.

We look forward to hearing your views. For more information, please visit our Substack https://polymathmind.substack.com/