Possibilities and limitations, of unstructured data

This article appeared, in full, in ESOMAR’s Global Market Research 2022 report 

The rapid growth of unstructured data offers many possibilities but also new challenges. As global firms need to manage this growing abundance of information, there are an increasing number of suppliers specialised in analysing unstructured data – many of them AI-based, technology-investment start-ups. We asked the experts what the role of humans is in the data-obsessed future. 

Managing zettabytes of knowledge 

According to Gartner, unstructured data represents an estimated 80 to 90 per cent of all new enterprise data. Furthermore, it's growing three times faster than structured data. More precisely, research firm ITC predicts that the volume of unstructured data is set to grow from 33 zettabytes in 2018 to 175 zettabytes by 2025 – that is 175 billion terabytes! Not only will most data be unstructured, but the International Data Corporation (IDC) also estimates that only about 10 per cent will be stored. Even less will be analysed. 

The zettabytes are dizzying indeed, especially for organisations that are increasingly relying on this unstructured data for various reasons, from regulatory and analytic purposes to decision-making. Unstructured data will be the driving force behind analytics, machine learning, and business intelligence. Over the next few years, it will deliver many business benefits but pose challenges as well. 

In a recent Venturebeat interview Krishna Subramanian, president and COO of Komprise, a data management software provider, said that enterprise IT organisations intuitively know a lot of their data is unstructured and growing in double digits, “but they don’t know exactly how much they have and how fast it’s growing.” 

Unstructured data, a description

There are various descriptions of unstructured data. This type of data cannot be stored within the neat, uncluttered columns of an Excel sheet. This is why it poses challenges when attempting to search and analyse this information quickly. Unstructured data can include conversations through e-mail or text messages, but also social media posts, blogs, video, audio, call logs, reviews, customer feedback, and replies in questionnaires. 

“In simple terms, unstructured data is all the information we can get to”, concludes Hana Huntova, executive director at SIMAR, the Czech Association of Agencies for Market Research and Public Opinion. “And perhaps it is unstructured only because we don’t know the fitting key to unlock it.” She expands on her description by taking a closer look at ‘text’, as it is used in media sciences and cultural studies. “Text is understood, approached and analysed in two ways: on the one hand, simply as a form of communication - for example, in film, articles, files, TV programs or sheet music. On the other hand, we can look at the interaction between the text and its reader; its effect on the reader, the possible methods of understanding the text and its content, and exploring the character of the content.” 

Michalis A. Michael is the CEO of DMR, a pioneering tech company in CXM using artificial intelligence for unstructured data integration and analytics. He agrees that text, audio, images, and video are all part of what we call unstructured data. “Another way to think of it is to consider the opposite, which is structured data; the numbers we find in tables or graphs. Various sources claim that more than 90 per cent of all humanity's recorded data, from the beginning of time, are unstructured. It is richer, it is qualitative, but only at scale.” 

Growing awareness

Whilst most organisations can manage their structured data effectively, the real challenge lies in unstructured data. After all, this is where the real context and insights can be found. Not surprisingly, 70 per cent of organisations will shift their focus from big to small and wide data by 2025, expects Gartner. However, many organisations drown in data, as it is accumulated every second by, for example, countless interactions with clients, reviews, sales, or social media comments. 

Huntova believes that the rise of unstructured data analytics is important because it has similar potential to its structured counterpart. “It has many forms; it can be approached from different angles, and each of us can explore different approaches and tools. On the one hand, we can look at the various data sources and debate endlessly what is data and what isn’t and whether it is structured or not. And then, when we look at which methods, tools, and approaches could be applied, the sky is the limit.” 

However, there is one thing we should keep in mind, she urges. “We need to keep asking an important question: how can unstructured data assist our mission to better understand consumers, society, and indeed the world in general? Does this data offer a better, more accurate, richer and more colourful understanding of the issues at hand?”