The Same Answer Is Not the Same Evidence

Abstract

Synthetic consumers are becoming increasingly capable of producing responses that seem plausible, coherent and surprisingly similar to those of real participants. But a similar response is not necessarily the same evidence. Synthetic responses can be valuable for prediction, exploration and hypothesis generation, while research with real people allows us to observe how meaning is actually constructed. As synthetic consumers become more convincing, understanding this difference becomes increasingly important.
 

The debate around synthetic consumers often revolves around a relatively simple question: do they work? Can they reproduce human responses, anticipate patterns, simulate segments or generate results sufficiently similar to those obtained with real participants? These are legitimate questions, of course. But perhaps they are no longer the most important ones.

Even if a synthetic response works extremely well, seems plausible and coherent, and even matches what real consumers may later say, a fundamental methodological question still remains: what, exactly, was evidenced? This shifts the discussion from the idea that synthetic data are “good” or “bad” to another question: what happens when a response is generated by a model rather than produced by a person participating in the research?

A synthetic response may accurately predict what a person might say, but that is not evidence that someone actually said, felt, thought, negotiated, remembered or experienced it. The distinction is not between a good response and a bad response. It is between inference and observation. And this difference matters especially in qualitative research.

Getting the answer right is not the same as observing a phenomenon

Imagine that we ask a synthetic consumer why they would stop buying a certain brand and the response is: “I would probably stop buying it because it no longer represents my values.” The statement is perfectly plausible. It could even appear in almost the same way in a later interview with real consumers. If that happened, we could say, with reasonable confidence, that the model made a good prediction.

But what actually happened, empirically, when the synthetic response was produced? No consumer recalled an experience or tried to explain a past decision. No one hesitated before answering or felt the need to justify a choice. No identity was negotiated in front of an interviewer, and no participant reconsidered their own thinking while speaking.

The system generated something that a certain type of consumer could plausibly say. This can be extremely useful, but predicting a possible response is not the same as observing the human process through which that response emerges. The same answer is not necessarily the same evidence.

What does a human response actually evidence?

Qualitative research has never assumed that consumers necessarily tell “the truth”. People forget, rationalise decisions, try to appear coherent, give socially acceptable responses, interpret the past from the present and, often, do not even know exactly why they made a certain choice. Even so, one empirical fact remains: that person produced that response, in that context.

Consider a participant who says: “Price really is not that important to me.” Twenty minutes later, in the same interview, she explains that she stopped buying a certain brand because the price increased by two dollars.

The first statement may not correspond to her behaviour, but it is still data. It tells us something about the way that person represents her own relationship with price. The second statement adds another layer. Now we have a contradiction between stated identity and reported behaviour and, for a qualitative researcher, this inconsistency is not necessarily an error to be corrected. It may be precisely the finding.

Perhaps price really is not the criterion that this person likes to imagine guides her choices. Perhaps the issue is actually perceived value, or perhaps the increase exceeded some mental price limit. Perhaps saying “I do not choose based on price” is part of the way she wants to see herself. The investigation begins precisely in this distance between discourse and behaviour.

Now imagine that a synthetic consumer produces exactly the same two statements. Textually, the responses could be identical. Methodologically, the evidence is not. In the first case, we observe a contradiction produced by a person within a research experience. In the second, we observe a model generating a plausible version of that contradiction. As text, they may seem identical, but they are not identical empirical events.

Qualitative research does not study only responses

This may be one of the most important distinctions when we think about the use of AI in qualitative research. Qualitative research is not simply a process of collecting statements. It investigates the production of meaning.

A human response is a situated event. It comes from a person who has a history, a memory, a body, social relationships, previous experiences and a particular position in relation to the situation being investigated. And the response itself may change while it is being produced. One question brings another experience to memory. An intervention by the moderator makes someone reconsider an initial explanation. Another participant’s comment provokes an association. A contradiction emerges, a certainty weakens, or something that at first seemed irrelevant begins to occupy a large part of the conversation.

The researcher does not observe only the final content of a statement, but also the way in which that meaning takes shape. This is why, in qualitative research, as I usually say, the path to the answer is part of the answer.

A model can simulate hesitation, ambivalence, doubt or change of position with extraordinary sophistication. But the simulation of this process and the observation of this process continue to be different sources of evidence.

So, what kind of knowledge does a synthetic response offer?

Certainly not “none”. That would be an equally simplistic conclusion. A synthetic response may indicate that, given certain information, learned patterns, parameters, instructions and the context provided to the system, a certain response is plausible.

It can help us anticipate possible discourses and objections, generate hypotheses, explore different interpretations of a stimulus and identify areas that deserve to be investigated. It can also be useful in preparing discussion guides, testing questions, the initial development of concepts or the exploration of scenarios. In this sense, synthetic data can be extremely valuable as a tool for inference and exploration.

The problem begins when a plausible inference starts to be treated as if it were empirical observation, and this shift can happen almost imperceptibly. A report may present a synthetic statement in exactly the same way it would present a participant’s statement. An AI-generated persona may explain in detail why it “prefers” a certain product. A set of synthetic respondents may produce extraordinarily coherent and convincing patterns.

As language becomes more sophisticated, it may become easier to forget what actually happened. Fluency does not turn simulation into occurrence.

Prediction and observation serve different purposes

Instead of asking whether synthetic consumers will “replace” real consumers, perhaps it is more useful to recognise that they produce a different type of knowledge. A synthetic response can help us explore what someone with certain characteristics could plausibly say. Research with a real person allows us to observe what happens when someone encounters a question, a stimulus or a situation. Both can be useful, but they are not methodologically equivalent.

The better synthetic consumers become, the more important this distinction becomes. When a response is articulate, coherent and highly convincing, this boundary may become less evident. The problem is not in using synthetic consumers. It is in asking them to support claims for which they were not designed as evidence.

Synthetic consumers can play an important role in research. They can expand exploratory possibilities, reduce costs at certain stages, allow rapid experimentation and help researchers develop more consistent hypotheses before entering the field. If a synthetic response helps us formulate a hypothesis, it has already fulfilled a valuable function. If it helps us discover a better question to ask consumers, it has too. And if it lets us explore alternative interpretations of a concept before an empirical stage, that has value too.

But claims about what real consumers felt, thought, experienced or meant need to be grounded in evidence generated by real people or observed from them.

The question we should ask

Perhaps the most useful question when faced with a synthetic result is not: “Does this response seem human?” Not even: “Does it correspond to what human beings could say?”

The most important question is: “What kind of evidence does this response represent and what conclusions does it allow us to support?”

Synthetic data can tell us a great deal about what consumers could plausibly say. Research with real people lets us observe what happens when they encounter a question, react to and interpret a stimulus, recall an experience, or construct meaning. Sometimes, the two paths may produce exactly the same sentence. But the same answer is not necessarily the same evidence.