Data Marketplaces, the future for Consumer Understanding?

Introduction

Discussion in conferences, webinars and articles about the future of market research often highlights the implications for research agencies and the efficiencies generated by new technology. These efficiencies include automating and standardising research, agile research, VR and AI applications. All of which enables us to do research projects faster and cheaper. 

Another phenomenon which can drive speed is to use what we already know. This has led to the enormous rise of Insight Engines like Market Logic, Lucy, Sharpr and Stravito. External sources are used for this. The focus is on traditional data providers like Nielsen Retail Panel, GfK household panel, Euromonitor, Mintel etc. Over the last years, we’ve seen several Data Marketplaces emerge. In these Marketplaces enormous amount of external data sources is available. 

We’ll explain how these external sources can be exploited for consumer understanding and what it means for the capability of consumer insights departments. 

Data Marketplaces are two-sided platforms intended to match data sellers with buyers and to facilitate and manage data exchanges and transactions. Data Marketplaces have a general purpose. And that is to trade any kind of data. This is different to more niche Data Marketplaces which focus on certain industries or certain types of data. Recently Personal Data Marketplaces have been popping up, which allow consumers to get paid for sharing their data on a consent-managed basis. 

Overall, Data Marketplaces enable easy access to third-party data sources and let organisations monetise their own data assets. 

Why are external sources not exploited? What is the issue?

As noted above, a lot of external sources are available. So why is the consumer insights department not exploiting these sources to understand consumers and generate actionable insights as much as they could? Especially at a time when interview fatigue is increasing. Wouldn’t it be better to limit interviewing to qualitative deep dives and use as many existing data sources as possible? 

Several reasons can be given:

• Consumer Insights Managers are often not knowledgeable about these sources

• Consumer Insights Managers might have doubts about the data quality. Too often, it happens that consumer data is only trusted if it’s been acquired according to known methodologies.   

• Consumer understanding is often done on an ad hoc basis. IT often doesn’t support ad hoc projects that combine external and internal data.

• Vast amounts of data require data science/machine learning methods to analyse. Consumer Insights Managers often don't have (sufficient) experience with these methods. They also often don’t know how to manage Data Scientists.

• Data Scientists are analysing the data sources within Companies, but only the ones they can access. They lack knowledge about external data sources and aren’t approached for consumer-understanding requests. 

But we think the main reason external data sources aren’t used as much as they should be is that their usage requires creative thinking as solutions for business issues. It requires, on the one side, out-of-the-box thinking and, on the other side, the capability to make connections to business situations. Two examples can highlight this: 

1. Edinburgh airport has several flight routes to London. They want to know the level of competition. Their competition consists of two railways:  the Railway Edinburgh – London and the Railway Glasgow – London, but Edinburgh airport didn’t know the number of travellers.  Which sources could they use to find this out? They approached a mobile phone supplier. They requested to deduct from the location-linked mobile data (and to analyse) the travellers by plane versus the two railways. 

2. Another example is from the beverage industry. An international beverage company wants to develop the cider category globally and is rolling out its main brand. The basic variant was launched, including two or three flavours, in a given market. An important research question for many beverage companies is which flavours to launch in which markets. Thus, the goal of this project was to develop a tool which would screen certain flavours upfront and check their potential in a given market. They looked at leading categories concerning flavours, such as chewing gum, tea and carbonated soft drinks.  Higher flavour volumes/shares suggest a higher potential for beverage category flavours. The assumption is that when more flavours are introduced of that certain flavour in a given country, the respective flavour will be more receptive and have a higher volume potential. 

Both examples need a different thinking framework. This capability is, in general, underestimated. Especially because when people see the solutions, they think that it’s logical and simple. But to come up with these solutions requires a capability which is hardly experienced amongst Consumer Insights Managers – being able to judge the quality of external sources. This is vital as e external sources might not cover the full picture.  

How do the data marketplaces operate?

Typically, a Data Marketplace charges a commission fee for the service of connecting the provider to the buyer. Buyers can pay in several ways like one-off data purchase with no rights to updates to the dataset, ongoing data subscription and a usage-based data license. 

To attract traffic, Data Marketplaces offer free data sources. This is open-source data made available by authorities or public institutions or also outdated samples of their products. This means that buyers can play around with them and get to know how useful the whole data product could be for their purposes before buying it. 

Buyers receive data in several ways. This can be via an API, or through a specific software or software platform as well Dataset bulk download. 

• The following Data Marketplaces have a significant presence Advaneo: offers: 2.000.000+ data products. Strong presence in open data made available by authorities or public institutions. 

• AWS marketplace: offers around 4.000 data products. Several Data brokers like Experian and Acxiom are trading their data.  

• Carto: Offers a location Intelligence platform. Their product Data Observatory gives access to thousands of public & premium spatial datasets and saves time on gathering, cleaning and analysing data.

• Crunchbase offers business technographic data to discover industry trends, investments, and news about global companies–from start-ups to the Fortune 1000. 

• DataRade offers around 3.500 data products. More than 2,000 suppliers, amongst others Acxiom and Gravy Analytics. Offer a significant amount of commerce and consumer sources (purchasing, audience, demographic etc.)

• DAWEX pretends to be the largest 2-sided Global Data Marketplace, gathering multiple thousands of data acquirers and data providers from more than 20+ industries around the world

• DIH offers around 45.000 data products. Strong presence in open data made available by authorities or public institutions.

• Knoema: 2.000.000+ data products, lots of public institutional data.  

• Refinitiv offers around 250 data products and is financially focused. This was previously Thomson Reuters.

• Snowflake: offer of 1.500 data products.. More than 240 suppliers, amongst others Acxiom, Gravy Analytics, Globaldata and Knoema.  All data sources are stored in Snowflake, and selected data can be downloaded. Instead of every user having an individual copy of the data, Snowflake allows multiple users to access the same copy simultaneously as long as they have the right permissions. This means that when a data provider sells a data product, it only sells access to their data rather than a copy of it.

• Veracity offers data sets and tools on an industry cloud platform. Its’ main focus is on maritime, oil and gas, energy and renewables. 

The Landscape is scattered and complex. It requires expertise to find your way in this data jungle! 

What should companies do to exploit external data sources for consumer understanding?

Companies can appoint a so-called Data Scout. A Data Scout understands the business issues but also has a Consumer Insights Manager track record. They’re knowledgeable and can assess whether external sources contain sufficient information for the business issue or if (additional) market research is still needed. They also understand Data Science. This means she can steer data scientists for specific projects and able to translate the results into business recommendations. Data Scouts are capable of developing relationships with Data Marketplaces and know how to connect external-data streams with other data sources while being an out-of-the-box thinker, as explained at the beginning of this article.

What do we recommend?

We recommend that companies start to experiment with how to harness information from external sources. Companies can start with a small project that requires external data and involve one external dedicated agency led by a Consumer Insights Manager with a data scientist.  It will create awareness and knowledge of Data Marketplaces in the company and for which kind of solutions these can be exploited.  It will also allow companies to evaluate the need and benefit of appointing a Data Scout on a structural basis. Companies taking the lead in harnessing external sources will be able to react faster but also obtain a richer and more cost-effective Consumer Understanding.