The essential dimension of evidence in research

The Insight250 spotlights and celebrates 250 of the world’s premier leaders and innovators in market research, consumer insights, and data-driven marketing. The awards have created renewed excitement across the industry whilst strengthening the connectivity of the market research community. Nominations for the 2024 Insight250 are now open at Insight250.com. 

With so many exceptional professionals named to the Insight250, it seems fitting to tap into their expertise and unique perspectives across various topics. This regular series does just that: inquiring about the expert perspectives of many of these individuals in a series of short topical features.

This edition features Jane Frost CBE, CEO of MRS (The Market Research Society). Jane and I discuss the growing role of evidence and its importance to research in an ever-scrutinising society that has waning trust. This is a deep conversation that calls upon Jane’s eclectic experience as a research expert.

Crispin: Jane, for the last 12 years, you’ve been the CEO of MRS and have long talked about the importance of evidence - what’s this about, and why is it so important?

It seems to me odd that so many people can think that so many vital decisions can be made without the evidence to support that decision. Mind you, in these times when the Post Office can ignore all the evidence of problems with their IT and continue to prosecute sub-postmasters for years, championing evidence is still necessary. Evidence does not have to be data; indeed, data doesn’t always tell the story. You need the context behind seeing gestures and hearing emotion to really understand the story of the evidence. So many data sources can mislead; they can let you know how well you are managing processes rather than outcomes, for example. Let’s not forget either how data can be collected that ignores swathes of the population or market, as Caroline Criado Perez so vividly wrote in her book Invisible Women.” 

Jane Frost

Crispin: Can you share some examples from your time as a client leader at Shell and the BBC (British Broadcasting Company) where evidence has proved commercially or socially crucial? 

I was well trained in brand management at Unilever in the importance of ad hoc created evidence from research to complement sales data. It was some inspired in-depth qualitative that finally convinced us that none of our positioning concepts was going to beat the power of Fairy in the dishwashing market, and made us do a deep dive into the users of that brand as opposed to merely the purchasers. There are examples strewn throughout my career where evidence from internal data and research programmes have enabled my team and myself to change minds and gather investment as well as save money! 

“The BBC was at one point determined to de-prioritise Children in Need; it was not deemed creative or original or stylish enough, and not one of the major programmes or channels gave it more than cursory support. At the same time, the BBC was failing to attract a C2DE-based audience to meet its claims of universality. My team analysed the audience data and showed that the event actually delivered an audience that the BBC never normally reached, and furthermore, it was an audience that participated. Children in Need has become a core BBC strand (oh, and I convinced everyone from Sir Elton John to Bowie to perform Lou Reed’s song, Perfect Day, for free to raise over £2.5m as well ).

“Without evidence, we all make decisions based on our own bias; in the Gulf, for example, translations into Arabic for a Shell Super Plus ad would not have identified that the right description for the brand was related to words related to the best upmarket food rather than technical specifications. That surprised the chemists.”

Jane Frost

Crispin: Is evidence used enough in the public sector/government? Can you share some examples of when you saw this work?

Government frequently forgets that evidence doesn’t have to be based on longitudinal academic studies that can take years. These absolutely have their place, and who would want to take a medicine that didn’t have the full gamut of this sort of testing? But the government works in short-term cycles related to the budget, the press narrative and elections. To stop or underpin decisions that are made in these time frames needs a different approach to research.

“For example, decisions made in the Home Office which create new criminal offences obviously have an impact on the Courts as they create more cases. Sometimes, criminalising an issue is not the right way to achieve the outcome everyone wants. To get women out of abusive relationships frequently, it is better to prioritise fast action and support (and yes, details like ensuring pets aren’t left behind are important ). Women frequently don’t want to criminalise their partner or their children’s father for emotional reasons. My minister understood these arguments intellectually, but it wasn’t until he watched an in-depth interview that he understood emotionally, and that made the difference in his personal commitment to the issue.

“In HMRC (tax), we showed that we could save tens of millions of pounds and get better outcomes for the public by cutting the amount of advice we gave benefit claimants – basically, there is a direct correlation between the amount of advice and the numbers of errors made - the more advice, the more errors. This, on the surface, is counter-intuitive. If you don’t believe me, those of you who have children must recall when they got worse marks because you helped them with their schoolwork.”

Jane Frost

Crispin: People talk a lot about insights, facts and data, but how can we be sure (or as sure as we can!) that decisions are being made based on the truth or on evidence? 

Bad research, whether it is because the desk data is flawed ( much internal data like sales are collected based on the way a company’s internal systems and accounting are organised, rather than customer reality) or the samples are inadequate or the questionnaire design is shoddy, is almost worse than none at all.

“Even good research based on unvalidated assumptions is dangerous. Think of setting up false choices for customers. Ask the public if they would like benefits to change to reflect their circumstances, and they will say yes; ask them if they would prefer that or guarantee their benefits to turn up on time. You get a different answer. People who need to budget need certainty of receipt of income as a priority. Research that relies on a weighted sample of 1 to be representative is obviously flawed – but it can happen. In these cases, you are missing the right to a voice for many sectors of the public, but you are also doing yourself a commercial disservice – missing market opportunities. The MRS Representation in Research Group has the research to prove that. For help on this and much more on Equality, Diversity and Inclusion in research, go to: https://www.mrs.org.uk/resources

“Furthermore, in the UK, market research is defined as a scientific process; this guarantees many of our operational freedoms; let’s not put that in jeopardy. “This is an election year in the US and the UK – you can guarantee that our methodologies will be under a lot of scrutiny.

“All parts of the research supply chain have a part to play in delivering ethical research that delivers an impact, clients as well as agencies and consultants need to engage in real depth from the very start in creating the brief and ensuring the quality of the outcome.”

Jane Frost

Crispin: People talk a lot about insights, facts and data, but how can we be sure (or as sure as we can) that decisions are being made based on the truth or on evidence?

The eagle-eyed amongst you will have seen that I have managed to answer most of these questions without mentioning the elephant in the room / the ghost at the feast / the pot of gold at the end of the rainbow ( delete according to your levels of optimism).

“The truth is, as a sector, we have been pursuing the AI path for a very long time now under a lot of different names. The new AI products will liberate us from some of the work that has been pretty much drudgery until now, and they should liberate us from spreadsheet doom. They should allow us to develop concepts at speed. There is a huge opportunity. The nature of the tasks we do and the talents we need will change. Some jobs will go, but Google and I agree new jobs will be created, such as in security and verification. To avoid the obvious pitfalls, more time and skill are needed in the briefing process as well.

“There are risks – I have heard of AI being used to generate fake audiences (to ensure they had enough women in an event audience). I know that it can be normative, currently embeds the risk of missing under-represented audiences and the bias of all the historical data on which it has been trained. Many clients are creating their own AI systems to avoid the risk to their IP of generic systems. Freely available systems risk being no more differentiable than existing search.

“But we are a sector that generates insights from evidence, and we should not be precious about the methodology used, only the outcome of our research and the ethics and standards we use to achieve those insights. The UK market has grown to over £9bn. Over a third of that revenue is export based, and I believe that is evidence to support the fact that the demand for the high-quality evidence-based insight we generate here will remain high.”

Jane Frost

Crispin: To succeed in championing evidence, what skills do we need, and how should readers gain those skills?

Every time we ask senior leaders client and agency side, we come up with the same key skills and talents required for successful insights”    

• Core data skills

• Storytelling ability

• Understanding of context, especially the business environment

• Creativity

• Flexibility

“There are two MRS Delphi (the think tank for MRS) reports that emphasise these skills: creating a customer-led organisation and Insight Alchemy (mrs.org.uk/pdf/insight_alchemist_final.pdf).

Jane Frost

Crispin: The MRS has published more on the importance of evidence. Where can readers find out more?

There are so many free resources to be found on the MRS website (mrs.org.uk), including client-side advice on inclusive research, guidance on the use of generative AI in research, reports from all parts of the research supply chain. I moan to anyone who listens about the need for researchers to be doing their own research using our own resources.”

Jane Frost

Hot Topic

AI is currently all the rage, but frequently, data used can be normative - how does this impact both diversity and inclusion and also generate “eureka” moments? Should we be worried?

The best response is to treat AI as you would any new product, examine its potential but treat it professionally and with clear eyes – don’t get sucked in by the hype. Think “privacy by design“ and MRS Code of Conduct principles – these will apply to AI as any other methodology. Then think again about representation, challenge claims and seek advice from specialists in the area of ED I research or from our website.”

Jane Frost

Top Tip

What’s your top tip to be a leader and innovator in our profession?

Leadership is a rare quality. It is emphatically not about management. Good managers can be poor leaders. Leaders develop a diverse ecology in their teams, and they do not need to be the tallest tree in the forest. They do need to care about their purpose and the people around them, and they need to take personal responsibility for outcomes. It can be a lonely place as you must model the behaviours you expect from others. So leaders inspire trust, shouldering responsibility for failure and sharing success. They liberate people around them to be their best.

“It sounds impossible, but it starts with respect for those around you. Cultivate respect for your team and your customers and spend time outside your organisational bubble, learning from anywhere and everywhere. Develop skills outside the day job. Try becoming a mentor and shadowing the CFO. Learn the “language” of others and never stop remembering that any relationship demands work. We should be good at much of that as researchers.” 

Jane Frost

Crispin: Jane, thank you for a fascinating conversation - it is clear why you are an Insight250 Legend, and I have no doubt that your insights will reinforce all our views that “Evidence Matters”. Thank you.