You CAN make sense of the nonsensical

If your company is sitting on vast amounts of text data, you’re not alone. With open-ended survey questions, customer reviews, social media chatter and more, there are a lot of conversations going on that can help you understand your audience. However, all this text often seems overwhelming. It is perceived as difficult to process, can appear disjointed, and contains language that is messy and confusing - and this can be exacerbated depending on the industry or category in which you do business. These challenges can’t be solved with traditional word clouds or sentiment rankings. 

Taking a trip

So how can you tame language that seems untameable? We took a trip to answer this question. Our team put their heads together to think of where we could access the most nonsensical text data available, we landed on people describing drug trips. With recent media attention on the use of psilocybin mushrooms to treat certain illnesses (on many U.S. ballots in the midterm elections earlier this month) and shows such as Netflix’s How to Change Your Mind about the history and uses of psychedelics, this topic seemed appropriate and timely. In fact, economically, the value of psychedelics is increasing rapidly, with a Research and Markets report estimating that the market will grow from $4.75bn in 2020 to $10.75bn by 2027. 

When it comes to making sense of the language of a psychedelic trip, the first thing we needed was the words themselves. To obtain them, we conducted a survey of 4,000 people. The survey featured a screening question asking whether participants had ever tripped after taking LSD or magic mushrooms (psilocybin). We then asked closed questions to gather demographic data (age, gender) – although we did not ask for any other personally identifiable information given the sensitivity of the subject. After asking participants to select which substance they’d tried, the final question was open-ended, and simply asked respondents to describe their trip experience in as much detail as possible. 

And the survey responses we received were just about as messy as it gets. Here’s just one response:

“A quasi-synesthesia takes form as my “i”, my internal sense of ego or being, becomes the music playing from above me in the loft. The sounds of the music intertwine with my thoughts in a beguiling way. These sounds and my “me” manifest as immersive closed eyed visuals. Magnificent teal and mint-coloured globules slowly float through a black abyss.” 

What did we find? 

Before AI-assisted text analysis, it would have been impossible to make sense of trip reports like these. Respondents’ streams of consciousness are impossible for humans to wade through, make sense of, categorise, and draw conclusions from. However, using our comparative text analysis platform, which leverages natural language processing, we could take this enormous data set of 5 million words and surface the key differences between multiple data sets - resulting in a gold mine of information after just a few seconds of processing.

For example, we found that women were 9.9x more likely than men to discuss emotions to describe their trips, using words like ‘passionate’, ‘sadness’, ‘fear’, ‘happiness’ and the like. Men underplayed what they felt and were 5.8x more likely to use the word ‘fairly’ as an adjective, which shows a desire to dampen how much the experience may have impacted them. We also examined the differences in how people of varying generations discussed their trips. Millennials were 4.1x more likely to highlight mental health as a reason for taking drugs, as well as being 4.5x more likely to say that it was a form of ‘therapy’ for them. Constrastingly, Gen X was more focused on treating physical pain and, unsurprisingly, Gen Z was just looking to have a good time. 

This kind of data is critical in creating a holistic picture of your audience, diving in deep to understand the “why” behind what they are doing, buying, downloading, clicking on and more. So if you think making sense of your text data is an insurmountable task, this study helps prove that even the biggest data set with the most fantastical language can be processed, analyzed and help you make critical decisions. Using the right technology, and the power of natural language processing and comparative text analytics, you can make sense of even the most nonsensical text data.