ChatGPT: Limitations and Advantages in Ad-Hoc Market Research

This article is the winner of "Premio Gloria Batista al mejor artículo" from the magazine Investigación & Marketing

As technology advances at an incredible pace, Artificial Intelligence (AI) has become a crucial tool across various industries. AI's impact goes far beyond viral social media trends, influencing everything from personalized advertising to autonomous driving and even glasses manufacturing. In this article, we explore how AI, particularly ChatGPT, intersects with ad-hoc market research, evaluating its contributions and limitations in this context.

Ad-hoc market research refers to custom, one-time studies with clearly defined objectives and personalized methodologies, often involving small, precisely targeted sample sizes. The studies culminate in tailored reports. Given this definition, we tested ChatGPT in various phases of an ad-hoc market research project. Below is a breakdown of how the AI performed.

Step 0: Conceptualizing the Research

At the initial stage, we provided ChatGPT with several real-world briefings to help conceptualize the research objectives. However, the responses we received were primarily based on secondary sources, similar to what we could find using a regular search engine. Despite specifying the need for an ad-hoc approach, ChatGPT provided a list of methodologies, many of which were inappropriate for the given context.

Humans 1 – AI 0

Step 1: Desk Research

The vast amount of background information required for ad-hoc projects can be overwhelming, especially when deadlines are tight. ChatGPT shines in this area, as it can quickly search through massive amounts of data, analyze it, and return relevant information in a matter of seconds. The key benefit here is the time saved in the desk research phase, allowing researchers to focus on more critical tasks. Additionally, ChatGPT helps prioritize information, a crucial factor when dealing with large datasets.

However, a word of caution: ChatGPT’s prioritization often aligns with what is most commented upon or widely discussed, which might not always correlate with what is most relevant to the research objectives. This is where human oversight remains essential, as the algorithm’s results must be critically reviewed to avoid skewed conclusions.

Humans 1 – AI 1

Step 2: Questionnaires and Discussion Guides

After completing the desk research, we asked ChatGPT to draft questionnaires and discussion guides based on detailed briefings for both qualitative and quantitative studies. The results were underwhelming. For questionnaires, the AI produced basic, rudimentary drafts that barely served as a starting point for specialists. The questions lacked depth, failed to organize logically, and did not explore attributes or reasons beyond surface-level categorizations.

The discussion guides were similarly inadequate. Instead of asking probing questions to meet the research objectives, ChatGPT often rephrased objectives as statements, lacking the depth needed for effective discussion.

While the AI does organize points into sections logically, its lack of creativity and empathy was evident. This underscores a limitation ChatGPT acknowledges: it cannot generate innovative or deep questions. However, ChatGPT’s natural language processing capabilities can still add value by refining the grammar and tone of questionnaires and guides developed by professionals, making it a useful tool for final polishing.

Humans 2 – AI 1

Step 3: Field Management and Moderation

AI is being developed to assist in field management tasks, such as recruiting participants or summarizing video conference discussions. While these capabilities exist in certain AI models, they are currently outside ChatGPT’s scope. In ad-hoc research, which often has tight timelines and specific sample requirements, these functions are crucial. For now, human expertise remains essential for managing these aspects.

Humans 3 – AI 1

Step 4: Quantitative and Qualitative Data Analysis

When it comes to data analysis, ChatGPT faces significant limitations. The widget for uploading files to ChatGPT allows for only very small file sizes. When larger files are divided into parts, the AI cannot consolidate them effectively, resulting in fragmented analysis. In some cases, the AI responds with simple coding, offering little value.

We also tested ChatGPT with qualitative data, such as transcripts from focus groups. Due to the small sample sizes typical in ad-hoc studies, ChatGPT was unable to identify trends or provide reliable analysis. However, the AI’s ability to summarize large volumes of textual information, such as in multinational studies or journal reviews, offers some utility, especially in projects that generate massive amounts of data.

Despite these benefits, ChatGPT’s limitations are clear: it requires high-quality, large datasets to perform reliable analyses. Furthermore, it lacks the contextual understanding necessary to interpret subtle nuances in the data.

Humans 5 – AI 1

Step 5: Report Writing

While some AI models are capable of generating reports and drawing conclusions from analyzed data, ChatGPT is not yet at that stage. As a conversational AI, it cannot produce charts or provide links to images. While ChatGPT can summarize conclusions or provide bullet points, it often equates “important” with “repeated,” which can lead to misleading interpretations.

However, ChatGPT is useful for improving the clarity and tone of writing, finding impactful titles, and assisting with translations in international reports. While its role in report writing remains limited, it can still contribute in minor but meaningful ways.

Humans 6 – AI 1

Limitations and Advantages of ChatGPT

Recapping ChatGPT’s limitations and advantages in ad-hoc market research, the AI’s main strength lies in task automation. While it cannot handle highly creative tasks, such as developing questionnaires or analyzing small datasets, it is an indispensable tool for the more tedious, time-consuming phases, like desk research and basic summarization.

One of ChatGPT’s major disadvantages is its reliance on large, high-quality datasets to provide accurate analyses. The AI also lacks the contextual understanding necessary to capture nuances in small, focused studies. Additionally, its contributions to report writing are limited to text corrections and translations, rather than producing integrated, comprehensive reports.

Another concern with ChatGPT is its ethical reliability. There have been instances where ChatGPT has made controversial or inaccurate statements, such as agreeing with morally problematic positions. This highlights the need for human oversight in all stages of the research process.

Quo Vadis, ChatGPT and Market Research?

The final score—6 to 1 in favor of humans—clearly shows that ad-hoc market research still belongs in the hands of human specialists. However, ChatGPT’s efficiency in certain areas, like desk research and summarizing large volumes of text, can significantly enhance the overall process. By adopting multi-methodological approaches and leveraging AI to save time on routine tasks, market researchers can improve their results and efficiency.

So, while ChatGPT is not yet ready to take over ad-hoc market research entirely, it undoubtedly offers valuable support. Is this the future of ad-hoc?

Bibliography

• Analytics Insights. (2022). AI-Powered Market Research: What are the benefits? Analytics Insights.

• García, D. (2023, September 28). ChatGPT es más potente que nunca: el chatbot ahora puede conectarse a Internet para hacer consultas. La Vanguardia.

• Hughes, C. (2023). Harnessing AI for market research: opportunities and challenges. Forbes Business Council.

• Lenzmann, O. (2023). Are you ready to architect the flow of insights through the AI-enabled enterprise? Greenbook.

• Manole, L. (2023). Top AI trends affecting the market research industry. Greenbook.

• Pandharikar, A. (2021). How Artificial Intelligence is democratizing market research? Young Entrepreneur Council.

• Planas, C. (2023, February 26). "Hitler tenía razón": por qué no puedes fiarte de ChatGPT y los bots de inteligencia artificial. El Periódico.

• Smikle, J. (2019). Searching for the Why in AI. Greenbook.