The quality problem changed. Our tools didn’t.

For much of the past decade, the market research industry made real progress on data quality. Device fingerprinting, attention checks, speed controls and post-survey cleaning helped online research scale while limiting obvious fraud. These techniques remain widely used today, and they still play an important role. What has changed is not the presence of quality controls, but the nature of the risk they are expected to manage. 

Increasingly the most damaging quality issues do not come from blatant automation or easily identifiable bad actors. They come from respondents who appear legitimate on the surface, pass technical checks repeatedly and still behave poorly once they enter a survey. AI has accelerated this shift by making low-effort responses harder to distinguish from thoughtful ones at a glance. The result is a widening gap between what looks valid in the moment and what proves reliable over time.

When legitimacy stopped being the same as trust

Most quality tools were designed to answer narrow, well-defined questions. Does this device appear legitimate? Is this response internally consistent? Did this participant trigger a known rule? Those questions still matter, but they were never designed to answer the harder question research teams increasingly face: can this person be trusted to behave reliably across studies?

That distinction may be subtle, but it is becoming central to how quality failures actually occur. Today’s problems are less about individual bad responses and more about patterns that emerge across participation. Viewed one survey at a time, it may be that nothing appears obviously wrong. Viewed across studies, suppliers and months of activity, the risk becomes more clear. This is why quality can feel harder now, even for teams using more safeguards than ever before. The checks remain optimized for evaluating moments, not behavior over time.

A familiar analogy helps clarify the gap. A single transaction tells you very little in isolation. A credit score, by contrast, reflects patterns of behavior accumulated over time. It does not predict what someone will do today with certainty, but it offers a far more reliable signal than any one transaction ever could.

Why point-in-time checks keep missing the same problems

Consider a participant who appears technically clean across dozens of observations. Their device remains stable, their technical scores consistently sit near the top of the range and nothing about their setup raises concern. That picture changes once behavior history is taken into account. Across those same observations, the participant has been removed for quality reasons repeatedly, flagged multiple times for disengagement and produced only a single clean complete. No individual survey would surface that pattern; it only becomes visible when behavior is examined across time. Seen through a historical lens, this respondent no longer looks like a generic “clean” complete, but maps clearly to a recognizable behavioral persona defined by declining trust.

As participation becomes more distributed and respondents move fluidly across platforms and suppliers, these kinds of patterns are becoming easier to observe and harder to ignore. Traditional quality workflows discard this context at the end of each project. Respondents are evaluated again and again as if they are new, even when their past participation suggests otherwise. Quality becomes something that is inspected after the fact rather than understood at the source.

The tools themselves are not failing. They are doing exactly what they were designed to do. The challenge is that the environment they operate in has changed in ways those original assumptions never accounted for.

Why managing quality at the source matters

Other industries learned long ago that quality does not scale when it is treated as a cleanup exercise. Manufacturing improved reliability by identifying defects early rather than correcting them at the end of the line. Supply chains became more resilient by verifying sources instead of relying solely on downstream inspection. Data quality follows the same logic. 

When issues are detected only after fieldwork is complete, the damage has already occurred. Time has been lost, budgets have been spent and confidence in the data has been undermined. The earlier risk can be identified, the less expensive and disruptive it becomes to address.

Managing quality at the source does not mean eliminating every bad response. That has never been realistic. It means reducing avoidable failures by prioritizing the signals that are most predictive, not simply the ones that are easiest to collect. In practice, this requires moving beyond point-in-time checks and incorporating history. It requires understanding not just whether someone looks legitimate right now, but how they have behaved across real research environments.

Practical steps teams can take now

Teams can begin addressing these challenges by reframing the questions they ask, rather than by overhauling their tools or workflows. Instead of focusing solely on whether respondents pass technical checks, consider how quality is evaluated over time. Look for ways to retain behavioral context across projects rather than discarding it at the end of each study. Examine quality patterns at the supplier level, not only within individual surveys.

Most importantly, distinguish between legitimacy and trust. Device-level signals indicate whether a setup appears valid. Behavioral history indicates whether participation has been reliable. Both matter, but they answer different questions.

This is where the idea of a trust score becomes useful. Not as a silver bullet, but as a way to make history actionable. Much like a credit score compresses complex financial behavior into a signal people can use, a data trust score provides a practical way to assess reliability before data is collected, rather than relying exclusively on cleanup afterward. 

The quality problem has become more complex as the data environment has changed. Until tools and infrastructure reflect how participation actually works today, quality will continue to feel reactive, fragmented and harder than it needs to be.