It gives a high-level assessment of the survey design. Usually, the pilot has a small sample, but looking at the respondent experience metrics qualitatively helps to improve the questionnaire design and the experience on a project basis. The output will suggest the improvement actions.
4. Post-live experience
After exploring the respondent, creating a relevant design and measuring the pre-live experience, it is a time to assess your effort. Establish a tracking program to get the respondent feedback after every live interview, this type of assessment should be very short and includes the “RXM” (illustrated above in table 1) which will allow you to have a “respondent experience index" for each project.
Apply the following formula to calculate the Respondent Experience Index (RXI):
RXI Range (-100 : +100)
E: effort (% of “easy” + “JR” responses)
C: clarity (% of “clear” response)
L: length (% of “JR”+ “Shorter than expected” responses)
B: burden (% of “yes” response)
5. Building a espondent experience (RX) database
While the tracking program (post-live experience) is running, create an internal database where you can store the respondent experience metrics along with the respondent experience index for each project (the post-live experience will feed the respondent experience database). Additionally, make sure you add the profile fields in your database, so the fields would be:
Geographical coverage (city/ region/ country)
The planned length of interview (Est minutes)
The actual length of interview (Avg)
The actual length of the interview (Min)
Actual length of interview (Max)
The incentive given (Y/N)
Data collection method (CAPI/CATI/CLT etc)
Effort score (% of “easy” + “JR” responses)
Clarity score (% of “clear” response)
Length score (% of “JR”+ “Shorter than expected” responses)
Burden score (% of “yes” response)
It is important to use any database application that can help you to set search criteria and retrieve the data easily.
Now the processes start to reach an end, after exploring the respondent, measuring the real experience and building the respondent experience database; we need to bring the respondent experience in the research to life, applying the following actions supports this stage:
Firstly, before proposing any research design refer to the RX database to gain understanding. Search for any previous project which seems similar to the one in the proposing stage, compare the respondent experience metric and index with the implemented design (i.e. method, LOI) and then apply the learning to the new project. As you can see the model will work as an optimising tool for future research design.
Another benefit of applying the model is that after a period of time (e.g. year) you would have “a respondent experience index” on different levels (e.g. industry, data collection method, LOI). It could be an indicator of how to engage the respondents and get high-quality responses. It could be a unique selling point or differentiator across research providers.
Secondly, utilise technology. Explore and utilise technology that makes the interview easier and effortless. For instance, survey platforms could introduce a pre-loading feature [f] that reduces the length of the interview and respondent effort improving the overall experience
Beacon [g] is another technological tool, the applications of this technology in market research can vary. It could be a reminder to the respondent to participate in any research that focuses on a real-time experience in a specific location so that a tool can run an NPS transaction survey. This helps the researcher to get accurate responses by reducing the respondent’s effort in memorising the things around the experience. Moreover, it creates a good feeling and enhances the respondent experience.
Third, questionnaire gamification [h] as stated by Gamification Research Network -GRN [e], in the academic realm, several studies in various contexts have shown that gamification can be an effective approach to increase motivation and engage users or participants in a given activity.
The respondent experience has to be at the top of researchers’ priorities. Memorable or at least reasonable experience is a vital requirement to get quality of responses, deliver reliable insights and recommendations. ERMBI model is a trial to manage and achieve such an experience.
[e] Gamification Research Network, http://gamification-research.org/2016/09/how-to-gamify/ See e.g. Hamari et al. 2014; Morschheuser et al. 2016 for reviews).
Morschheuser, B., Hamari, J., & Koivisto, J. (2016). Gamification in crowdsourcing: A review.In proceedings of the 49th Annual Hawaii International Conference on System Sciences (HICSS), Hawaii, USA, January 5-8, 2016.
Hamari, J., Koivisto, J., and Sarsa, H. (2014). Does Gamification Work? – A Literature Review of Empirical Studies on gamification. In proceedings of the 47th Hawaii International Conference on System Sciences, Hawaii, USA, January 6-9, 2014.
[f] Pre-loading feature allows users to save known information for the target respondents before starting the live interviews (e.g. demographic variables) and then attach the same information with the respective respondent responses after the completion.
[g] Beacon is a tool to pinpoint the location of a portable device (smartphone) holder and then triggers (push messages or remainders) can be sent based on predefined criteria for the location.
[h] Gamification is a technique of using the games out of the context of playing. The core concept of gamification in marketing research is all about making the questions more “game-like” which can take several forms.