Diminished voices: Combating representativeness issues in Market Research

In 2016, the political polling industry was hit by a humiliation from which it still has not recovered. All the predictions were wrong: Donald Trump, not Hillary Clinton, was heading to the White House, and trust in polling plummeted.

Buckets of ink have been spilt trying to explain the miscalculations, but one of the most likely reasons was widespread inadequate sampling methods – particularly an underrepresentation of non-graduates. [1] 7 years on, polling has changed and improved, but, in some ways, commercial research is making the same mistakes.

How can we make sure that our research reflects the full breadth of the population we are examining? And whose voices have we been excluding?

The problem with panels

These days, almost all quantitative research is facilitated through online panel providers. Surveys can be sent to thousands of respondents in an instant, who can complete them wherever and whenever they like, making fieldwork far less expensive and quicker than before. 

But panels are, by their nature, unrepresentative. Unlike probability sampling – commonly used before the advent of access panels – panel sampling does not give everyone in the population an equal chance of taking part in a survey. 

The racial and class profile of most research panels falls far short of representation [2], with fewer voices from socioeconomically disadvantaged backgrounds. Consequently, poorer people and those from ethnic minority backgrounds have, at best, a disproportionately smaller voice in research which guides how products and services will be developed.

So, a lack of representation in online panels goes far beyond impacting research quality - it has been a factor in excluding voices from important conversations.

Promoting a more representative research world

As the MRS notes [3], the solution to this is to ensure that these groups are included and well represented through careful sample composition. However, social grade and ethnic background are not always captured on surveys, and when they are, they tend not to be included in nat. rep. quotas (age, gender, and region are often the only measures considered).    

As ever, the barrier to progress may be cost. Applying more quotas will often lead to higher sample costs, and with budgets tight, there is an incentive for all those involved in the research process to keep sample costs low. Panel companies are not going away: they will continue to be the primary means of sample recruitment, so the solution to the problem will be persuading research buyers that they should invest in representative samples. But how can we, as research sellers, do this? 

The most compelling approach is to frame this in terms of the benefits they will receive; a more representative research study will obviously lead to more accurate and more actionable results. So, this more inclusive approach need not be framed as pure altruism; firms which adopt better research practices will have a clear competitive advantage.

This can be particularly persuasive if framed within clients’ research objectives: segmentation groups will be more accurately sized and defined, understanding of usage occasions in category reviews will be improved, and innovation research will be more effective at identifying sources of volume and incrementality.

Researchers can also present this issue in the context of more established quota standards. Nobody with a passing understanding of research principles would accept a sample which is disproportionately older or overwhelmingly male; the justification for setting quotas on age and gender is widely agreed. Why, then, would firms be satisfied with research which disproportionately represents higher social grades or excludes those from certain racial groups? 

Additionally, most major research buyers are outwardly committed to inclusivity. Many have indeed committed to diversity, equity, and inclusion initiatives to ensure they are doing their part for a fairer world; surely, a key demonstration of this commitment lies in guaranteeing consistently representative research. A company genuinely dedicated to inclusion should seek feedback from all its customers. Appealing to inclusivity is unlikely to garner a negative reaction, even from large corporations.

Ultimately, change can start with us as researchers; by persuading our clients to adopt more inclusive sampling methods, we create incentives for attracting individuals from underrepresented communities to participate in panels. Advocating representative sampling provides a multitude of benefits: shaping business success and fostering a more equitable and insightful world for all.

Sources:
[1] https://academic.oup.com/poq/article/82/1/1/4837043
[2] https://www.focaldata.com/blog/5-ways-to-improve-consumer-and-social-research-with-ethnic-minorities
[3] https://www.mrs.org.uk/resources/representation-in-research-steering-group