A threat to the industry?

This article appeared, in full, in ESOMAR’s Global Market Research 2022 report. 

In such a fast-changing world, it’s hardly surprising that the way organisations buy and use insights services is also on the move. But the huge shift towards bringing projects in-house, and using tech platforms to deploy them, is causing some hand-wringing about how much of what we call market research can be done by non-researchers. It’s also raising questions about what might be left for agencies whose work has provided the foundation – and benchmarks – for good work in the insights sector.

Ray Poynter, in his role as co-developer of ESOMAR’s 2021 Users & Buyers Global Insights Study, has worked with ESOMAR and other partners to look at the proportion of research/insights work that’s being done in-house versus what’s being done by research agencies. 

In 2020’s research, nearly half of all research projects were being done in-house; that had risen to more than 50% in 2021, as the total number of projects being undertaken also went up. 

“The desire has been there a long time, but it wasn’t possible,” Poynter says. 

Ray Poynter is chief research officer at Platform One

Not only is the volume of research expanding, but so are the areas of an organisation that are commissioning and using it. 

 “It’s not as if the insights team is bringing it in-house and doing it all themselves. It’s more profound than that. The design team are doing it, the operations team are doing it.” 

Ray Poynter 

Simon Chadwick sees plenty of change in this sector and says common estimates of the volume of research/insights work being done by platforms and other tech-enabled players understate their role by a factor of three. 

“What we saw during the pandemic was, in many companies, the realisation that models we've had of consumer behaviour don’t work anymore. 

The first question they threw at customer insights was, ‘what the hell is happening?’ And that meant that they had to do real-time but scrappy research to be able to provide the answer, which actually fed the explosion of in-sourced, self-serve research.” 

Simon Chadwick is managing partner of Cambiar Consulting, a change management consultancy.

The next question, Chadwick says, was, “what’s going to happen?” – which needed more strategic research. Freeing up funds for this meant putting even more of the simpler work onto platforms, and then a lot of that strategic budget went not to research agencies but to management consulting companies. 

The businesses now involved in what we have traditionally defined as market research – but which don’t generally call themselves market researchers – are as diverse as they are plentiful. McKinsey & Co., PwC, Bain & Company, Salesforce, Adobe and Google. There are many, many more, as ESOMAR’s Global Top 50 Insights Companies for 2022 showed. 

Lower barriers, broader market 

One of the big players and more obvious direct rivals to established agencies is one of the driving forces behind organisations automating and in-housing their research. 

Samuel Bakouch says relative newcomers to the sector are sometimes rivals to the established players but often complementary service providers. 

“When you have a use case that’s heavily repeatable, like tracking your brand or doing a usage and attitude study, or collecting feedback on the market or from prospective customers, you're seeing technology platforms like ours really replacing a lot of what an agency would provide, with a better quality/speed/cost ratio,” he says. 

Samuel Bakouch, Momentive's head of product

What the self-service platforms have successfully done, Bakouch says, is blow up the idea that quality has to cost a lot and that cheaper services are, by definition, inferior. And that has lowered barriers to access to research and increased demand for the broadest definition of market research. 

A platform for expansion 

“Research and quality insights are being really democratised,” Bakouch says. “I think that's a really powerful trend in the industry, recently amplified by the pandemic.” 

Samuel Bakouch

Increasingly digitised businesses generate huge amounts of their own customer data, for instance. Such data streams can be a veritable goldmine of insights, but data has to be interpreted and viewed alongside so many other sources of information, not all of which is usually done in-house. 

Vanessa Oshima is the former head of marketing at Starbucks Japan, ex-VP of strategy at Coca-Cola Japan, and former GM of women’s sport at Nike Japan. She now heads up Heat-Data, her consultancy. 

She says that the rise of automation and self-service platforms has helped research clients. Still, the proliferation of data dashboards raises some big, important questions – that more established research agencies are well placed to help organisations answer. 

Chadwick says the wave of research tasks going to management consultancies now seems to be ebbing and established research has been making something of a comeback, but not just by doing more of what it always did. 

“I think what's happening is that as things calm down, and as decision-making has reverted back to heads of insight functions, they are using their established research supplies for more of this strategic work,” he says. 

Simon Chadwick

Many of the established players, meanwhile, are taking on a client role of sorts, using automation platforms and systems to make their own processes more efficient and effective. 

“(Clients) are going to look for research agencies that have really good research fundamentals but that also are strong on strategy and that do data visualisation at a very high level,” says Chadwick. 

“What was the established research market is going to mutate into offering a lot more than just research.” 

Simon Chadwick

Expansion into what? 

“It’s possible for the pie to get bigger and for agencies to not necessarily have an absolute loss,” says Poynter. 

Ray Poynter

From the research user’s point of view, Oshima says agencies need to think less about promoting the value of their proprietary widget, tool or dashboard. Rather, they need to be generous with their time in demonstrating what it can do, as well as humble about acknowledging that there are some things that others – agencies, platforms or individuals – can do better.

“Often, they just want us to work with just one agency, but the reality is not going to work like that again, so they need to be comfortable with the fact that they're a piece of the puzzle and be able to work across different agencies and tech providers,” she says. 

Vanessa Oshima

Overlapping services are not a threat to research, Chadwick insists. 

“Lines are being blurred – not only the lines between data analytics and research but also between quantitative and qualitative. They are increasingly being intertwined.” 

Simon Chadwick

Agencies should therefore look for benefits and opportunities in the tension between old and new, Oshima says. 

Agencies have to overcome the sense that they alone can be a kind of “mega mind” that has all the best answers to a client’s question. In reality, the most high-impact breakthroughs tend to come from a collection of insights used in combination by smart people (note: not AI-powered platforms). 

And when tools, agencies and individuals contribute to real progress, the results can provide strong evidence for further investment. So rather than fight for 100% of a small pie, why not have 25% of a pie that’s ten times the size?