Response biases (Source: Bogner & Landrock, 2016; Gideon et al., 2017; Krosnick, 1991; Roßmann, 2017)
The "lazy" online survey participants are thus to be distinguished from the "dishonest" participants who, due to social desirability, go out of their way to present themselves positively. However, their careless answering causes equally falsified answers, which are highly detrimental to the validity of studies.
Measures to reduce response biases
The likelihood that a participant exhibits response biases is determined by different factors, including the question complexity, familiarity with the survey topic as well as their motivation. Based on this, measures are developed that can be implemented to reduce response biases.
Reducing the question complexity
The complexity of a question is considered high if its focus, word choice, or sentence structure poses a challenge to steps of the response process, for example:
A question that contains low-frequency words (e. g. "discrepancies" instead of "differences") or has a complex syntax is difficult to comprehend and can be understood differently.
Retrospective questions that use an inappropriate (e. g. number of everyday products purchased in the last two years) or ambiguous time frame (e. g. whether a restaurant was recently visited) make it difficult to recall information and make judgements.
Response options that are not overlapping cause difficulties during response selection.
When dealing with complex questions, participants are more likely to show response biases. For this reason, questions should be formulated simply and clearly. This measure has the advantage that market researchers have full control of the question complexity – in contrast to factors on the part of the survey participants. However, depending on the study interests, questions cannot always be simplified without oversimplifying them.
Screening of participants who are familiar with the survey topic
Participants who know the survey topic well are likely to have already formed opinions on this topic prior to the survey. The effort to recall existing memory content is significantly less than for others who have no to little experience with the topic and have to assess it spontaneously. As an example, it is certainly easier for users of a product to evaluate it than for someone who knows it from hearsay. Thus, they are less likely to show response biases.
Against this background, response biases can be reduced if participants are selected based on their familiarity with the topic. This measure is easy to implement using screening questions but is not always in line with the study interests or the representativeness claim of some studies.
Increasing the participant’s motivation
The more motivated the participants, the less likely response biases are to be observed. For this reason, various measures are developed to increase their motivation, which show empirical effects against different response biases.
One of these is asking survey participants to promise to provide honest answers or to verify that was the case. This results in a lower average number of nonsensical answers to “bogus items” (e. g. agree to the statement “I have never brushed my teeth”) – an indicator that respondents show less bias to arbitrarily select an answer option. An increase in response consistency can also be ascertained.
In other studies, participants are warned that careless responding has consequences for them. This measure has the same effects as calls for commitment. In addition, warned participants are less likely to straightline and spend more time on the survey.
Incentivisation also reduces the “mental coin-flipping” response bias and increases the completion time. Memes, i.e. funny photos with motivational captions, do not help diminish response biases.
The described measures do not counteract all, but only certain response biases. It is also worth noting that warnings or calls for self-commitment can leave a negative impression. They are therefore not long-term solutions and should only be implemented with caution and after weighing up the advantages and disadvantages.
Response biases pose a risk to study validity that must be taken into account. Given the research interest and other conditions of the studies, market researchers should implement appropriate measures against this source of error. The role of data cleaning to capture participants with response biases should also not be underestimated.
The article is based on Hieu Nguyen's own research and opinion.