Plus ca Change in the Insights Industry?

Your perspective on this question will perhaps be contextual - your length of time as an insights practitioner; your client and/or agency experience; your specific role or function.

But there’s no doubt that the business of scientifically-driven market research which I entered in the early 80s (yes, 80s..) is very far removed from the AI-accelerated, platform-based, norms-underpinned, and heterogeneously served business of insights that we now inhabit.

There are probably 3 themes to address here -  WHAT has changed? WHY have changes come about? In what ways are these for the BETTER (or worse)?

So, let’s review things using the following lenses:

1. Methodologies

2. The evolution of the Clientside function

3. The Insights characteristics which separate the Winning Clientside Companies  from the Losers

4. Insights usage – how have businesses evolved their consumption and usage of Insights?

5. Industry structure

 How we do what we do

Firstly, some obvious seismic changes: quantitative face-to-face data collection now only accounts for  6% of interviews globally, and as little as1 to 2% in markets such as the USA and UK.

Second – we don’t even bother with consumers at all for some types of investigation. Or, more accurately, we do, but we rely on accumulated consumer permissioned databases such as Kantar’s Link and apply LLMs to create predictive algorithms.

Some businesses (for example, Colgate-Palmolive)  are now partnering two AI systems in an integrated process – one to surface unmet consumer needs and another to develop new product concepts against those – but perhaps the key point here is not that they are doing this but that they have a clear Human Guidance component built in. So, Tech is again an enabler (as were punch card readers in 1982!) not an “owner”. HumInt will seemingly always play a role – possibly in a more balanced way – with the proportion of total task hours taken by thinking being more than in the past, thanks to the removal of process hours?

And third, the smart application of everyday technology means that even ethnography, in which as researchers we actually lived with people for short periods, has now become a remote activity. Importantly, much of this is now “self-serve” rather than gathered or observed, with consumers providing UGC and commenting on each others’ submissions. Welcome to the world of   co-created insights.

Clientside – more change than agencyside?

I’d argue yes. This is not to denigrate the immense analytic and delivery strides made by agencies but the bigger delta is to be found in client CI functions..

Since I presented a paper to the MRS in 2008 with Verity Clifton (then of GSK), in which we set out some potential avenues of evolution for the clientside Insights function, we have moved a lot. My first exposure to clients in 1982 was of a market research department, staffed exclusively by folk with stats or social science degrees, sitting in oak-panelled rooms, hand-editing questionnaires which the agency had mailed to  them(!) a few days before, and waiting for some gilded youth from the elysian field of brands to turn up with an out-of-the-blue order to execute a brief. And their role was to serve, not to question, interrogate the brief, educate their client or – heaven forbid – suggest the need for research..

They were executors, scientists of one stripe or another, experts dedicated to the art of measurement. Future foresights and trendspotting were peripheral activities at best. And as for joining-up insights across projects, or having any idea how the Business made profit, that would have been the exception not the rule.

The more inspired Insight leaders of the time (and there were many) set out a vision in which their teams would become business-literate insighters: welcomed to the Board, consulted at all stages of the business process.  The themes below are good representation of their dream.

And has that happened?
Mostly yes.

At one of my major clients, for example, all Insights personnel must at some point cycle through both the Trade Marketing and Brand functions – not as interns but fully fledged team members. That means they have an expert grip on issues as diverse as what the sales team know about their ultimate consumers, how the route-to-market works, how much profit they make per SKU and so on – all of which makes them best-in-class business partners.

So, this is a mindset change. But again, fuelled and accelerated by Technology. CoPilot’s ability to synthesize in minutes a raft of complex reports combined with self-serve platforms with embedded AI tools mean that by lunchtime you can have empirical feedback on the hypotheses which Copilot presented to you earlier in the morning.

It ultimately does need a human to steer the narrative to the specific business question and land it in the right way, so that is a constant. But a further change is that there is talk in some quarters of needing fewer humans in the team to do that – up to 30% fewer according to some estimates. So, more – or rather more targeted – work, with lower headcount.

 Winners vs losers

Which brings us onto what now defines the Winners and the Losers?

There’s a very useful and simple framework to use to understand this. Kantar Consulting conducted a huge research exercise amongst clients in 2016 which uncovered the 10 key drivers of an effective Insights Engine. The punchline here is that it wasn’t the artisan skills so valued in the past but knowing the business and being able to tell stories for a senior non-research audience. An inspection of the outer ring of the framework amplifies this – these are business skills, not research skills – but where research is the “rare earth mineral” you are working with.

This was then followed up by a study in 2021  called Insights 2030.

In this work, adopting the hypothesis that clientside functions had moved on and embedded the requisite business skills, the aim was then to tease out the specific lived behaviours which separate the Leaders from the Laggards.

To do this, they set the clients’ own assessments of their Insights teams against their financial results – in effect correlating Insights characteristics and behaviours with RoI. The resulting model showed the ROI gap between Leaders and Laggards to be significant in the majority of instances.

Those who have seen and seized the need for change are winning. They listened to what their stakeholders wanted and how they wanted it to be brought to them.

And they instilled business skills.

Also in this study, the Leaders cited “working with cross-functional teams” to manage data-to-insights 25% more often than did the Laggard companies. This is key -  they recruited more diverse teams and taught them how to integrate (see Matthew Syed on the problems of groupthink and the empirical proof of the value of diverse teams in “Rebel Ideas” and elsewhere).

They trained their people to be more business-literate: they shortened (everything – questionnaires, meetings, powerpoints, debriefs,) and most important of all they integrated insights.

Steve Silvers, quoted in the ESOMAR 2024 report, speaks to this “You’re never going to get all the data you need yourself. You’re going to have to stitch together these disparate datasets and augment your data with other data. Flexibility is going to be key – a Swiss Army Knife approach to measurement”.

This is arguably the #1 requirement. To exemplify, with one of our clients we produce their Category Growth Trajectories, utilising a host of data sources owned, acquired via ourselves and other agencies and publicly sourced (e.g., economic barometer inputs) to arrive at a Business narrative which covers all the growth opportunities and challenges, and with accompanying analyses of all available strategies. Furthermore, the data behind all this is available on a self-serve platform to run “what-ifs”. To do this, we put our people into their team – true integration.

So in many ways the clientside function – aided and abetted by their agencies – has delivered on the asks I shared in my 2008 MRS  paper, summarised in one of the client quotes as “Research needs to become a fusion of advice, analysis and prediction – in a business context”.

And this speaks to about the issue of Insights usage.

The function has elevated itself from the dark days when one Global Insights lead described the approach of client functions as:

“Recessive, reactive and content to be a support function. More comfortable with managing research than driving business outcomes. Backroom data crunchers, answering questions posed by other business leaders.” And perhaps in today’s environment, fatally “Too slow to act – wanting to have all the answers rather the amount they need to mitigate risk or to experiment”.

In addition to re-orientation of their role (often forced rather than welcomed – and the same can apply to agencies), what else changed?

Well, as with the endless argument around innovation as to whether an innovation succeeded because it was “pushed” by manufacturing advances or was a case of a brand recognising the zeitgeist and niche trends (see “From Marginal to Mainstream” by Helen Edwards), the change has been wrought both by Technology and by EQ.

Which is to say that the doing of research has now become telescoped. Clients can either DIFM (Do It For Me) – just email a few loose questions to an agency who will come back with a ppt deck in short order or, if you are more hands-on, just log on to an agency platform (there are literally dozens), upload your concept/creative, select an audience and your answer is back within a few days – or hours if you’re using the AI version. And within agencies our drag and drop process now also does the data work and charting for you. All of which means – more time for thinking, having the time to challenge, to ask “what if” not just “how much”, and ultimately to deserve that seat at the top table which was surely what every Insights Conference from the 80s onwards called for. And now we’ve got it. 

Take a look at the graphic below, courtesy of BCG and Cambiar, which perfectly exemplifies 2 things:

1. The sheer scale of the Insights “transformation” – even 10 years ago Cambiar was suggesting only 20% of companies were in the right place. Today, research done by Cambiar for GRBN suggests this figure is now north of 30%.

2. The mindset and skillset an Insights leader needs to have at her disposal nowadays. 

And finally, what IS this “industry” we talk about? What’s changed there?

This is a short coda really. Back in “the day” there were only market research firms. Or Institutes as they were often known, because the scientific focus was so great – it took decades before Insights businesses became known (even wanted to be known) as agencies. And that’s important, because agencies do what it says on the tin – they act to provide specialist services.

Two points to make in this context:

1. Traditional Market Research firms now account for substantially less than half of the Industry’s value @ $54bn out of a total of $142bn, (ESOMAR definition – 2024 report), with the increased focus on data analytics and wider intelligence meaning that this proportion is set to continue its relative decline, albeit in the context of a YoY growth of c4%.
   
   Consolidation and a shortening of “the tail” continues – the Top 20, buoyed by their IP models, ability to invest in R&D, and marketing clout, now account for close to half of overall spend, yet even here half of these have less than a 1% market share. So there remains an abundance of “micro” agencies, but the middle has been squeezed - many of the erstwhile independently owned mid-sized agencies have gone.

2. Do clients need agencies? Obviously yes, for access to IP-driven tools, expertise, outside-in and ex-category thinking – but so much can now be insourced and even R&D is easy with the range of tools and suppliers at a client’s disposal. As such it is very likely that we will see  a reframing of the relationship – one which provides many opportunities for both parties – is due very soon.

So, on reflection? Yes there has been huge change – in the nature of the enablers of insight; the reduction of doing in favour of the more valuable activity of constructing integrated business narratives; and in the breadth of intel which can be integrated. And all of this comes with significant shifts in the skillsets and types of staff required both agency and clientside.

What hasn’t changed is the core – the dedicated pursuit of revealing human truths, by humans.