The Reliability Gap: Why Market Research Needs to Ask Harder Questions of Its Data

The Insight250 spotlights and celebrates 250 of the world’s premier leaders and innovators in market research, consumer insights, and data-driven marketing. The awards have created renewed excitement across the industry whilst strengthening the connectivity of the market research community. Winners of the 202g Insight250 were announced earlier this month  - you can see the full list of Winners at Insight250.com.

With so many exceptional professionals named to the Insight250, we regularly tap into their expertise and unique perspectives across various topics. This regular series does just that: inquiring about the expert perspectives of many of these individuals in a series of short topical features. 

With technology advancing at an incredible pace and the value of insights ever increasing, I sat down with Insight250 Winner Jordan Peugh. As Chief Business Officer for SSRS, Jordan has extensive leadership and governance experience on the supplier side, with a specialisation in sales, client services and research. She leads the SSRS Sales and Client Service teams, directing research development, innovation, and strategy.

Crispin: You’ve suggested that many market researchers may be accepting samples they know are insufficient for their needs without pushing back. How widespread is that problem in your experience - and why do you think it has been allowed to persist for so long?

JP: I think it is fairly widespread, and two things are driving it. One is the longstanding “fit for purpose” mindset. The other is the rising cost of high-quality research. In many market research settings, clients are not looking for gold-standard population estimates; they want useful insight that can help inform a decision. In that context, fast and affordable research can be entirely appropriate.

The challenge is that demand for more data has grown even as the quality of many lower-cost options has declined. Some researchers and clients have not fully reckoned with how much quality has declined. Others understand the tradeoff but are not convinced better data is worth the added cost. So the issue persists for two reasons: speed and price are real business pressures, and the industry has become more comfortable with data-quality uncertainty than it often admits.

Crispin: What does it actually mean to understand and communicate the reliability of your data? What are the questions researchers should be asking of their vendors that they currently aren’t - and what should they do with the answers?

JP: It starts with asking more of sample providers. If you are using a panel, the basic questions are straightforward: How are panellists recruited? How long do they stay on the panel? How is fraud identified and managed? How much fraud is typically detected in a study? Those are not academic questions. Even modest levels of bogus respondents can bias estimates [1], and in some online environments the problem can be much worse [2].

The Esomar questions for online sample providers are a very good place to start. But asking the questions is only the first step. Researchers then have to look honestly at the answers and decide what those answers mean for the credibility of the data. Do the recruitment methods support face validity? Does the level of fraud raise concerns about representativeness or accuracy? If you cannot explain why the data reflect the population you care about, then you have to ask a harder question: are they useful for the decision you are trying to make?

We also need to be prepared to have real, honest conversations with our clients. They often are coming to us wanting research that is fast, cheap, and good. But that is increasingly hard to deliver with declining response rates and increasing prevalence of entities gaming the system. We have to explain clearly to our clients the reliability issues in various approaches and what we are doing to handle them.

[1] Bell, A. M., & Gift, T. (2023). Fraud in online surveys: Evidence from a nonprobability, subpopulation sample. Journal of Experimental Political Science, 10(1), 148–153. https://doi.org/10.1017/XPS.2022.8

[2] Bell, A. M., & Gift, T. (2023). Fraud in online surveys: Evidence from a nonprobability, subpopulation sample. Journal of Experimental Political Science, 10(1), 148–153. https://doi.org/10.1017/XPS.2022.8

Crispin: There is a difference between data that is unreliable and data whose limitations are understood and communicated honestly. Is the real issue not the quality of the data itself but the industry’s reluctance to be transparent about what it can and can’t tell you?

JP: I think it is both. Transparency has to come first, and researchers should expect it from their partners. But transparency only matters if we are willing to deal with what it reveals. Sometimes I suspect people do not push hard enough because they know the answers may be uncomfortable.

If transparency reveals serious concerns—very high levels of suspected fraud, for example, or signs of deeper quality problems—we cannot simply disclose that and move on. We have seen cases where high-profile findings later had to be withdrawn, including when YouGov retracted findings from its 2024 Bible Society study after identifying fraudulent respondents [3]. Transparency is essential, but it is not enough on its own.

[3] YouGov. (2026, March 26). Conclusions of investigation into 2024 Bible Society study. https://yougov.com/en-gb/articles/54406-conclusions-of-investigation-into-2024-bible-society-study

Crispin: You’ve suggested AI is bringing the reliability question even more front and centre. In what specific ways is AI making the transparency problem harder - and are there ways in which it could actually make it easier?

JP: AI is making data quality harder in two main ways. First, fraud is becoming more sophisticated, and that is the more serious challenge. Second, legitimate respondents may increasingly use AI tools to shortcut the process, which could affect the authenticity of the responses we collect. We are still learning how big that second issue will be, but it is clearly something researchers need to watch.

At the same time, AI can help identify suspicious behaviour and improve fraud detection. In that sense, it is an arms race between fraud and fraud detection. Further, on the topic of transparency, AI increases the complexity of what researchers need to disclose, monitor, and understand.

Crispin: What are some practical ways market researchers can adopt stronger data quality techniques from social science research while still working within real-world budget and timeline constraints?

JP: Market researchers might want to think more seriously about shared resources such as probability-based panels and omnibus approaches, including those offered by SSRS and others in the industry. Those methods can be efficient and comparatively affordable while preserving stronger scientific principles and offering better protection against fraud.

The use of benchmarking to assess data reliability is also a useful approach. Market researchers could benefit from integrating benchmarking from appropriate sample frames into their research process. If you are researching a population where there are decent federal benchmarks, make sure to evaluate your data against those to detect deviations, potentially even using high-quality data to weight or calibrate your samples.

Researchers should also be thoughtful about when qualitative methods are the better answer, especially for hard-to-reach or highly specific populations. In many cases, it is more valuable to get credible insight from a small number of well-matched participants than to collect 1,000 survey completes of uncertain quality. Market research does not need to replicate social science in every respect, but it can borrow more of its rigour around quality and adapt it to practical constraints.

Crispin: You’ve just completed your year as the 79th AAPOR President. What is the single most important thing the public opinion and survey research community needs to get right in the next few years - and how well placed is it to do so?

JP: It is important to remember that we make a deal with our respondents--they answer questions from us, and we accurately represent their wants, needs, and fears. When we let unreliable data into our studies, we are depriving our real respondents of their voice.

Beyond the business reasons to be vigilant for quality, as an industry and as members of society, it’s important for us to safeguard the public’s trust. To support this, we need to keep updating our standards, guidelines, and resources to reflect the realities of research today, including AI and the rising cost of obtaining valid opinion data. At the same time, we need to continue advocating for transparent and reputable research. That means refining our guidance, incorporating new risks into our standards, and staying serious about protecting public trust in the data we produce. I think the field is well positioned to do this, but only if we remain proactive. AAPOR is helping lead that work through its Transparency Initiative and its latest report, Responsible AI Integration in Survey Research [4], among other efforts.

[4] Rothschild, David M., et al. “Responsible AI Integration in Survey Research.” American Association for Public Opinion Research, 2026,https://aapor.org/wp-content/uploads/2026/05/Responsible-AI-Integration-In-Survey-Research.pdf

Crispin: Market researchers have always known that not all samples are equal, that not all data is reliable, and that the limitations of a study are rarely communicated as clearly as the headline findings. For years, that has been an inconvenient truth that the industry has managed to live with. AI is making it structurally impossible to ignore. As synthetic respondents, AI-assisted analysis and automated fieldwork enter the mainstream, the gap between what the data says and what it actually means is widening. Is the market research industry about to face a reliability reckoning - and is it ready for it?

JP: I think the industry is ready for that reckoning, but only if it is willing to look hard under the hood. This includes having frank conversations with clients and discussing reliability and the actual work that goes into providing good data. It also might involve occasionally pushing back against a client who has bought into the AI promise of ‘faster, cheaper, better’ without critically looking at what is actually being delivered. That also means investing in better data when better data are necessary, being more honest about limitations, and recognising that if a decision is not important enough to support quality research, then data may not need to play as large a role. That is not a comfortable message, but it is an important one.

Top Tip: We ask all our Insight250 contributors to share one practical recommendation with the ResearchWorld community. What is your top tip for a market researcher who wants to get serious about data reliability and start asking the right questions of their vendors and their data?

JP: My top tip is to ask one hard question before you field a study: if the data are unreliable, will it actually change the decision you make? If not, this may not be a place to spend money on research. If it will, then reliability and transparency have to be treated as essential. As a supplier, I know it sounds strange to say that sometimes the right answer is to do less research. But I would rather support work where the decision is made.

Crispin: Jordan, thank you for such a candid and thoughtful conversation and for sharing your expertise and insight, particularly around sampling, quality, and the increasing role of AI. These are all issues that so many researchers and insight professionals are facing head-on. You have challenged researchers to ask harder questions of their vendors, to benchmark their data against credible sources and to be honest with clients about what "fast, cheap and good" can realistically deliver. That challenge could not be more timely. Your point that unreliable data takes away the voice of genuine respondents is one every insight professional should keep in mind, particularly as AI raises both the risks and the stakes. Your top tip stands out most of all: before fielding a study, ask whether the data will actually change the decision. Coming from a supplier, that is a refreshingly honest message, and it deserves to be widely heard. Thank you for your leadership, both at SSRS and during your year as AAPOR President, in helping the industry face these issues with the seriousness they demand.