21st Century Insight: Change & Knowledge Hierarchy Permanence

The insights industry is dynamic and undergoing another technological revolution with artificial intelligence, but in a time of change it’s important to ask what doesn’t change. Artificial intelligence will certainly transform how we analyze information, but there are at least three things that will not change – (1) our focus on and curiosity about humans and their need states, (2) the basic mechanics of the market, and (3) the knowledge hierarchy. Leaders in the field should consider each of these, especially the last.

We can debate the immutability of “human nature”, but it is undeniably clear that the human is the unit of analysis in the poorly-named “market research” discipline. In the final analysis we are not studying a market or marketing. We are studying humans, their needs, desires, and behaviors. We may even study clusters of humans, either as a target market or in the B2B context, but the human is the indispensable core of what we study. That won’t change. Consider that the insight industry has evolved dramatically over its roughly 90 year history. It transitioned from door-to-door data collection, to landline, to internet. It globalized. It harnessed a quantum leap in computing power and data processing. It moved from an interrogatory-heavy approach to a more balanced approach utilizing text analysis and observation. And, as many of us forecast in the Journal of Advertising Research in 2011[1], “research in 2021 will represent a continuous, organic flow of knowledge—a “river” of information.”  The industry made all of these leaps, but the human was always at the center.

In the near future we may leverage more “synthetic sample”, bots trained to behave like humans. But even this sample will be designed to mimic actual humans. It will need to map back to real-life human opinions and behavior. We will also study how to influence consumers’ digital agents, their AI assistants. But, even then, the focus is ultimately the human. As I noted in the epilogue of Leading Edge Marketing Research: 21st-Century Tools and Practices[2], born in the industrial era with significant constraints on data collection, market research was naturally over-focused on the process, as opposed to the human.[3] In the 21st century we have attempted to correct for this with a focus on “insight” and human behavior.

Similarly, the basic mechanics of the market are unlikely to change. Humans will pay for products and services and Insight professionals will work to facilitate and increase these transactions. Humans and products are foundational. There will be new products, but they will still be products designed for humans or groups of humans.

This leads to the third thing that will not change, the knowledge hierarchy, sometimes referred to as the information pyramid, knowledge pyramid, or DIKW (Data, Information, Knowledge, Wisdom) pyramid[4]. Like commodities become finished products, value increases as data is refined up the knowledge pyramid. At the base of the pyramid is data. These are recorded observations or signals. This data is then structured into information. Information can then be studied, understood and absorbed as knowledge. And this knowledge can then be applied to action as wisdom. All insight professionals do the hard work of pushing data up this DIKW pyramid until it reaches wisdom. This is why so many firms and insight functions reference “data to insight.”

Everyone seems to have their own version of the DIKW pyramid. Some alternative versions of the  pyramid flow from data to information to knowledge to insight to strategy. We may never have consensus on this, and I’m not sure it matters. What matters is the pattern and the idea of taking observations of our world, structuring them, analyzing them, and turning them into something of great value.

The history is important here. While Russell Ackoff[5] and Milan Zeleny[6] are generally considered the fathers of the DIKW pyramid[7], it is worth noting that many trace the idea to T.S. Eliot and the chorus in his play The Rock[8], in which he asks:

Where is the wisdom we have lost in knowledge?

Where is the knowledge we have lost in information?

This highlights the interplay that has always existed between the science of market research and deep insight from the humanities.

Four questions drive this process from data to wisdom. The first question is “what?” What did we learn? This question pushes data into information. It gets us to the basic facts. The second  question is “so what?” What are the headlines? What is the actionable knowledge? This question moves the analysis from information to knowledge. The third question is “now what?” This question pushes the analysis from knowledge to actionable wisdom. Milan Zeleny, a knowledge management expert credited as one of the fathers of the DIKW pyramid, referred to wisdom as “why do” and simple information as “why is.” And the final question is “what next?” What can we reasonably anticipate about tomorrow? How can this tell us about alternative futures?

This leads us naturally to the future. The discipline and industry of insight is reorganizing itself in the age of artificial intelligence. Using the four levels of the DIKW pyramid, we can assess AI’s impact on the value creation process.

Data. AI agents are and will act as data collectors, both in observational research (sifting through unstructured text) and in interrogatory research (with bots asking questions). AI-driven qualitative research at scale may be one exciting application.

Information. Data processing is already heavily automated, and this will continue. Basic reporting can already be automated, but the information set must be highly routinized. This is why AI has been so successful reporting on baseball games and financial reports. AI can structure data and report basic information.

Knowledge. This is a much steeper climb for AI. It is built on information, but includes context. As Milan Zeleny put it, it is “know how.” We can assume that Artificial General Intelligence (AGI) will  be capable of processing knowledge. But that will require the creation of AGI first, and time estimates for its creation vary widely. 

Wisdom. For now, at least, wisdom is also the exclusive purview of the human insight professional. That could change in time, if we achieve AGI and co-evolve with it. I wrote a novel, Lincoln 2.0[9], exploring this potential future and how an AGI candidate (a revivified, AI Abraham Lincoln[10]) might interact with its campaign team and voters.[11] What I came to realize in the process is the enduring legacy of the Frankenstein story, the idea that we create a thing and it tries to destroy us. But what if we create a thing and it helps us?

Artificial intelligence will certainly revolutionize the data and information levels of the DIKW pyramid. It may even inform the knowledge creation layer in time. But the insight professional will sit atop the pyramid at wisdom. This has profound implications for the insight discipline. Fewer, radically fewer, employes may be needed in the data and information layers. Those process jobs can and will be automated. But more professionals will be needed at the knowledge and wisdom levels, and they will need to possess the skills we have often highlighted at conferences and in publications like this: advanced pattern recognition skills, the ability to synthesize across data sources and fields of knowledge, and the gift of strategy. These will require polymaths. They will require one last thing, intense curiosity about the world and other people.

And this leads us back to the beginning and our focus on and curiosity about humans and their need states. The technology will evolve, but the humans will remain the focus and curiosity will remain the driving force.

[1]https://www.researchgate.net/publication/233379923_Guest_Editorial_The_Shape_of_Marketing_Research_in_2021

[2] https://sk.sagepub.com/book/edvol/leading-edge-marketing-research/back-matter/d473

[3] https://sk.sagepub.com/book/edvol/leading-edge-marketing-research/back-matter/d473

[4] https://en.wikipedia.org/wiki/DIKW_pyramid

[5] https://en.wikipedia.org/wiki/Russell_L._Ackoff

[6] https://en.wikipedia.org/wiki/Milan_Zeleny

[7] https://www.researchgate.net/profile/Rob-Keller/post/Original_paper_of_From_data_to_wisdom_by_Ackoff_1989/attachment/63f67d8997e2867d5081d0de/AS%3A11431281121841684%401677098376991/download/Ackoff89.pdf

[8] https://www.poetrynook.com/poem/choruses-%C3%B4%C3%A7%C2%A3the-rock%C3%B4%C3%A7%C3%B8

[9] https://www.us.mensa.org/read/bulletin/features/lincoln-2-0/

[10] https://www.huffpost.com/entry/lincoln-20_b_8023744

[11] https://campaignsandelections.com/campaigntech/the-implications-of-an-ai-presidential-candidate/