The Advancement & Impact of Insights - An Insight250 Winners Series perspective with David Smith

The Insight250 spotlights and celebrates 250 of the world’s premier leaders and innovators in market research, consumer insights, and data-driven marketing. The awards have created renewed excitement across the industry whilst strengthening the connectivity of the market research community. Winners of the 2024 Insight250 were announced this past September - you can see the full list of Winners at Insight250.com. Nominations for the 2025 Insight250 close on March 31, so be sure to submit your nominations at Insight250.com.

With so many exceptional professionals named to the Insight250, it seems fitting to tap into their expertise and unique perspectives across various topics. This regular series does just that: inquiring about the expert perspectives of many of these individuals in a series of short topical features. 

This edition features David Smith. David is the Founder & Director of DVL Smith which provides training on insights and analysis. He is also the author of The High Performance Customer Insight Professional and has been awarded the MRS Silver Medal and ESOMAR Excellence in Intelligence Award. He holds a Ph.D. in organizational psychology and is a Professor at the University of Hertfordshire Business School.

I sat down with David to discuss the evolution and advancement of insights from his perspective which has spanned the industry for decades. It’s a fascinating discussion that reviews the insights industry from a unique perspective of this industry expert.

Crispin: You have been in the insight industry for many decades perhaps you could give us a quick update on where you’re now focusing your energy and efforts? 

DS: Well, my career has really been in three Acts. In Act One I learnt the insight craft primarily around the nine years I spent at what is now (NatCen) the National Centre for Social Research where I was privileged to learn so much from so many first class researchers about how qualitative and quantitative research worked.

In Act Two I founded my own business consultancy, built it up and then sold it to a large media group and ended up being the Chairman and CEO of what grew to be an international agency with offices in London, New York, Texas and Shanghai.

And in Act Three - where I am now - I’ve been focusing as a smaller consultancy on continuing to do insight projects for a range of different clients. But in Act Three I have also drawn together my entrepreneurial experience and also provide coaching and mentoring for smaller businesses - all based around my book The Entrepreneur Mindset -The Art Of Making Ideas Happen. And with the explosion of AI and the arrival of ChatGPT, I've become totally obsessed with how AI is going to change the insight landscape.

With my colleague Adam Riley we have written a book called the ‘Art Of Collaborating With AI: Harnessing The Power Of Diverse Thinking’ which will soon be published by Routledge. 

Crispin: In your career insight what were the big game changers that you’ve experienced?

DS: One way of answering this would be to look at the game changing technical developments that have happened since I first came into market research. You have to remember that I started my career being able to using IBM punch cards and soon after that I had my own Pet Commodore personal computer with 64k capacity and now we have AI - so the technical developments have been staggering. 

In addition there are of course the fundamental transformational methodological transitions. The obvious ones are the move from face-to-face to telephone interviewing to online research, which the industry by and large successfully handled – although maybe when it comes to some online research we perhaps need to revisit the rigour around the response rates that we are now achieving.

This may seem like a snipe at modern times but you have to remember that back in the day at Social and Community Planning Research (NatCen) we run surveys using probability sampling methods where we aimed at achieving a 75% response rate! (This may have meant doing up to 3 face to face callbacks on a respondent in order to secure an interview). 

Of course going forward in the hard-nosed commercial context that modus operandi could not possibly be sustained as economically and practically viable. But notwithstanding the need for pragmatism, perhaps we’ve slightly taken our eye off the ball in terms of not looking more closely at the issue of the representativeness of some of our samples. 

But the on a positive note the really important change that has happened in the course of my career in insight is the way that, on balance, we’ve moved from being data providers to being acknowledged as trusted partners when it comes to advising on understanding customers and helping organisations make informed customer-centric decisions. 

There’s always more we could do to raise the profile of market research and really drive home why insight professionals are so important to the success of organisations - and why we have some very special skills. 

But to my mind there has been a big shift in the course of my career towards recognising the role of insight professionals as business people able to grasp the commercial realities, add creative value and improve the bottom line. So I have seen a shift from us being seen as data suppliers aka ‘vendors’ (a term I always disliked) to being commercially aware respected business partners - which is very rewarding to see and report. 

Crispin: If you had to pick out three outstanding moments in your career, what would they be?

DS: Well I guess when you start a business in your spare bedroom and build it up, the first milestone is to hit that first £1 million turnover mark. We did that in about three years from a standing start without any outside investment using our own financing: so that was a really big moment – a target met, debt free and on course to treble that pretty quickly. 

Today in the UK still only a small percentage of businesses hit this £1 million target so I vividly remember achieving that as a big day all those years ago. 

But actually thinking about it, the entire entrepreneurial journey from the start up to corporate sale is collectively a kind of stand-out memory in its own right. So fresh in my memory is the move to our first office - a tiny little back room in an ice cream parlour in Epping, before moving to offices in Stratford in East London and then onto swish Shoreditch offices - which a few years later figured in a Victoria Wood and Julie Walters TV Christmas Special sketch. Apparently they wanted an office that communicated something creative, arty and slightly pretentious (I’m still not quite sure what to make of this.). 

And I guess the next big moment would be when John Wiley published my book ‘The Art And Science Of Interpreting Market Research Evidence’, which was a pioneering book on promoting the idea of taking a holistic view to the way we should be combining secondary, qualitative and quantitative data.

It was based on this book that I think ESOMAR awarded me the then John Downham Award for Excellence in Marketing Intelligence and it was a privilege to receive the award as John was an absolute hero. He was one of the leading market researchers of his generation - an ambassador for the industry given his senior and influential role within Unilever. 

Over the years I’ve been privileged to receive two ESOMAR best Congress paper awards, together with a number of other best Conference paper and presentation awards from ESOMAR, the MRS and other organisations. But without question the Excellence award was the standout highlight. (It came with a beautiful individually designed, hand crafted ‘trophy’ plus there’s an envelope stuffed full of euros! (Those were the days !). 

And I guess my other highlight would be a composite of all the high points in the 10 years or so I spent being the Convenor of MRS Winter and Summer schools attended by so many young and talented researchers, where we in those days we spent a week each year together down in Brighton or Eastbourne, having lots of fun whilst homing our craft skills. 

Crispin: You say you’ve become totally obsessed with AI over the last few years - in summary, where do you see this taking us ?

DS: On the day ChatGPT was launched I knew it was going to transform the life of knowledge workers - including insight professionals.

Soon after, with my colleague Adam Riley, and we started giving talks about the likely impact of AI on society and on the world of knowledge workers and insight professionals. At the beginning we weren’t exactly booed off this stage but I think the audiences were thinking these guys are over egging AI - surely it’s not going to make that much difference. Then over the last year or so we’ve got much more acknowledgement from our audiences of the fact we all do have to get our heads round how AI is going to change society - and quickly.

There will be the ‘newly advantaged’ who can massively benefit from now having almost a ‘second brain’ into which they can now tap. They can now outsource their thinking and get AI to do some of the heavy intellectual lifting. This will allow them to excel with their audacious creativity and deep insights into the human condition. But in the AI era there will also be the ‘newly disadvantaged’ - those who were expecting, given their qualifications and skills, a pretty successful career, but who may now find they need to pivot in order to acquire the fresh new skills they will now need to compliment what AI can now deliver.

For Adam and myself our response to AI has been to set up The Polymathmind Consultancy where our focus is not on teaching people how to use the different AI tools and apps. This is the easy bit. So we are about identifying the human skills that we will need to dial up in order to successfully collaborate with AI. We see these as being akin to the skills of the modern day polymath: being able to see the wider context, adding insights drawn from taking an interdisciplinary perspective on the problem and being able to apply that original human voice and creativity that is still beyond AI.  

This is the theme of our book on the art of collaborating with AI, and also the substance of different keynote talks, training courses and webinars we are now giving around the world on this topic. Those who are interested in some of the themes were developing might want to check out our Substack (https://polymathmind.substack.com/) where we’ve got lots of summaries of what the AI experts and pundits are saying, together with our own framing of the best next steps for insight professionals wanting to work alongside AI.

In summary, I believe as insight professionals we need to accept that going forward ‘average is going to come for free’ (and fast). And that as an industry we will need to enhance our creativity, forward planning, wisdom, emotional intelligence, compassion - all those silky skills that still allude to AI. 

Crispin: Do you feel ESOMAR and the MRS are doing enough to promote how we as an industry are now working alongside AI in this new era? 

DS: To be totally candid I feel that our industry bodies could be doing more to showcase how the insight industry will fit into the overall AI era. Everything is happening so fast so we can’t really be too critical but the reason for our recent ESOMAR Congress paper in Athens was to trigger more of a debate about how we position the insight industry - given the fact that AI will soon be undertaking vast swathes of the insight craft. I think one of the reasons why it’s difficult to get our head round how AI will impact our industry is because of the way AI advances. This has been described as the ‘jagged frontier.’ AI moves into sectors in an uneven way, identifying areas where it can go beyond just enabling a human job function to suddenly being an acceptable substitute for human input - it will provide in a few seconds really professional scholarly work.

But in other areas AI will still be performing in a fairly sketchy way that still needs very closely human monitoring. So do not see complete industries ‘falling’ to AI overnight - it is all happening by stealth in a softly, softly selective way. So it means as AI agents move into an industry sector - such as the law, accountancy or our own industry - it becomes difficult on a day by day basis to get a clear overall reading of what is likely to happen over the longer term.

So it is easy to fall foul of what it’s called Amara’s law - which is with technologies like AI we tend to over-hype their introduction at the beginning, but also still manage to underestimate their impact over a 5-10 year period.

And the other tricky thing about understanding AI is the debate often centres around the point will AI, if ever, become sentient. But this is missing the point. What we should be looking at is the point at which AI will be able to do 80% of what we do 80% as well (at virtually no cost and in a few seconds). This is the maths that will drive the rate at which organisations make the decision to substitute AI for people, rather than just staying with AI to enable a human process.

So it is very tricky to get your head round the speed with which AI will advance and the impact it will have.

But I do feel this should be a priority for the ESOMAR Council and the MRS Executive: they in my humble opinion need to better support their memberships given the massive changes ahead.

The good news is we as an industry have so much to offer and this is why I was promoting the idea in Athens of dialling up our ‘noble purpose’. I genuinely believe is our strength and where our future lies. 

In this way people would be clear about how insight professionals are constructively working with AI, but at the same time dialling up the human advantage we offer given the skills I’ve been discussing. So the noble purpose we floated in the Athens paper - A starter for ten strawman for discussion - was ‘we are diverse, interdisciplinary, polymathic thinkers who - in collaborating with AI - add creativity, wisdom and humanity to what AI can now do for us’. The copywriters can play around with this. We felt that we as an industry have such great skills that we should set ourselves a big prize and go for it!

Crispin: Given all the changes we are facing would you recommend a career in insight to young people?

DS: Absolutely, it’s a wonderful opportunity to be in at the ground floor of the new era of customer insight and business consultancy. 

The insight industry has always been a haven for people who are curious about understanding human behaviour and who have that ability to simplify and solve complex problems - those who also have the skills to create powerful evidence-based narratives that can bring about change. It’s always been an industry that’s been fun to be part of with a great network of fellow professionals there to provide support, together with there being lots of opportunities to train and develop. None of this heritage and legacy of the insight industry is going away with AI - in fact AI is going to heighten the need for individuals with the superpowers that will provide humans with that advantage over and above AI.

Insight professionals are well placed to play a key role in the AI era. We have our ability to make sense of complexity and apply advanced critical thinking skills. Then there is our ability to be ‘time travellers’ and understand what has worked and has not worked in the past and transports this into a vision for the future.

And there is our ability to be audaciously creative, together with being able to construct really compelling narratives - all set in the context of now being fluent in AI capabilities.

And we have the flexibility to adapt our leadership style to dial up the importance of the compassion and empathy that will now be needed in the AI era as people are facing rapid change and inevitably some anxiety about how they make the transition into the AI era.

So in short, insight professionals should seize the opportunity to be the industry - profession - that is at the forefront of how our brains will be evolving and developing as we work with Ai. This journey into co-intelligence - human AI collaborative working - is such an extremely exciting space to be in. 

TOP TIP

DS: Yes, to relish the fact that you are living at a massive inflection point in the history of mankind.

AI is going to change the structure of everything: the nature of work; our fundamental approach to education and the core fabric of society - and here you are able to shape how we do this for the greater good.

As an insight professional our business is understanding what makes people tick and how the human brain works, and how we improve the quality of decision-making. What better profession could there be to be part of than one that is right at the centre of the whole human-AI-Symbiosis debate?

The world needs insight professional who understand human nature, can work with evidence and who have the intellectual ability to see both sides of an argument. We need people who can provide that care and compassion around the way we arrive at informed decisions making maximum use of AI, but still retaining our own human creativity and original voice and in everything we do.

In sum insight professionals have a critical role to play in making sure we end up in the AI era with a fair society, where we use AI to build productivity and growth not to polarise society.

You, you can play out your contribution to this from a number of different industry perspectives, but for me the insight industry is right up there at the top of the tree when it comes to being a profession which understands human behaviour and is well placed to work out how best to creatively work alongside AI.

Crispin: David, thank you for providing such a fascinating perspective into your long-term career and the insight you have gathered from it. The ever-expanding role of insights is creating incredible opportunities for understanding and innovation, so getting your front-row perspective is fantastic. Thank you.