The state of social intelligence: 2020 year in review

Although it seems like a lifetime, it was only a year ago that I published my 2019 social intelligence year in review. Back then, I observed that individuals and organizations were finally discovering the unique role that social data, in concert with other data sources, can play in helping to solve real business problems.

While the uptake in digital data sources by  the research industry was well underway, little could I have imagined that a global pandemic would soon accelerate these dynamics, as almost overnight the world moved online. As one research agency friend put it: “Covid-19 probably condensed three to four years of market research evolution into three or four months." And as brands have become accustomed to these new approaches to conducting research and gathering insight, it’s hard to imagine a rush back to the more traditional resource-heavy, set-piece approaches of the past.

So, as we exit this most unusual of years, what are the main trends driving the awareness and adoption of social intelligence today? In my conversations with social intelligence practitioners, four themes emerged that are changing market research more broadly, thus presenting an opportunity for social intelligence to shine. 

1. Speed to insight 
Clients are pushing to reduce the use of traditional, ad hoc market research projects in favour of quicker, more cost-effective approaches that provide real-time, behavioural insights. Speed to insight has always been a strength of social data. Not only does it give researchers instant access to millions of public consumer opinions on just about any topic imaginable, accessing that treasure trove of insight is a fraction of the cost of traditional research methods with similar sample sizes. Furthermore, it’s reusable, with multiple projects able to repurpose the same data sets in support of different research questions. If you are looking for speed and value for money, social delivers.

2. Explosion of external data sources
20 years ago, if you wanted market share data, you had a very small group of providers to choose from. Today, you can get this data from dozens of sources. And as the importance and value of data have grown, many companies are now monetizing their own unique data sets. Everyone, it seems, is a data provider. The challenge now is in choosing the most relevant data set for the problem at hand or the question being asked. And far from being lost among these myriad data sources, social data is carving out an important place for itself, chiefly as a unique window into real observed human behaviours, with several advantages over traditional insight gathering techniques.

First, its unprompted nature gets around one of the big challenges of research—the introduction of bias by the questions being asked. Secondly, social posts are sent in the moment, when the consumer is actually having the experience, therefore representing a much richer and authentic view of consumers’ true feelings on a topic, as opposed to asking them to recount that experience days or weeks later and out of the context in which they had it. This feature of social data should not be underestimated, for very little market research delivered today is conducted with that type of context in mind.

And finally, social data’s continuous nature allows you to move away from the one-and-done practice that has historically characterized market research projects and towards a longitudinal approach to data gathering. This allows you to see trends emerge over time. It also has the added advantage of allowing you to go back and analyze a topic retrospectively, uncovering views and attitudes preserved in time, forever.

3. DIY meets data integration
Historically, full-service research agencies provided both the data and the insights, but they are now increasingly seeing competition from their very own clients. The growth of DIY research platforms has allowed brands to insource market research capabilities such as survey creation and distribution. And brands have made data gathering and customer feedback collection an integral part of the user journey and experience, and therefore already know a lot about their customers. When combined, brands are better able to use their customers’ behavioural and transactional data to ask the right questions at the right times, thus providing even more relevant customer insight.

To date, social listening tools, a form of DIY research, have been the primary gateway for organizations to access social data. This is quickly changing as social data feeds are becoming increasingly embedded within a wide variety of solutions, ranging from Enterprise Feedback Management to digital data analytics. It’s this combination of social data with other data sets that is so promising and, with a little help from artificial intelligence (AI) and machine learning (ML), should provide unique new insights into how people think and behave.

4. Truly game-changing innovation
From the phone survey to the online survey and from social intelligence to artificial intelligence, new sources and techniques have emerged over the years that have improved the diversity and efficiency of data collection and analysis. They have also brought with them a host of new entrants into the industry, whether they be technology providers or tech-enabled specialist consultancies. The adoption of these type of digital analytics providers, for lack of a better term, has expanded the definition of the market research industry and doubled its size, now reckoned to be valued at $90 billion.¹

One of the big underlying drivers behind this has been a slow but steady move away from question-based research methods and towards observational methodologies such as neuroscience, behavioural economics and data science. Dave Soderberg, Chief Data Officer at Black Swan Data, touched upon this revolution in his excellent blog post on the end of question-based market research.

Data Science in particular, aided by advances in different types of AI such as Natural Language Processing, has allowed us to understand and classify vast amounts of unstructured data (such as social) in a way that was unimaginable even a few years ago. It’s this continued innovation that will allow researchers and analysts the ability to finally leverage social data as the world’s largest focus group. As Jeremy Hollow, Founder & MD of Listen + Learn Research succinctly put it: “Why are we struggling to recruit people to talk about things they’re already freely talking about on social?”

2021 and the slope of enlightenment
After a year like 2020, it would be foolish to make any predictions about 2021 (as I attempted for 2020), other than to say that the importance and adoption of social intelligence within the insights world will only continue to grow. Even Gartner now thinks it’s finally entering the “Slope of Enlightenment.” It has certainly been a journey, with social intelligence reaching its formative, teenage years just as market research hits a midlife crisis and tries to redefine itself. So where do we go from here then? I’m reminded of the Royal Society’s motto, “Nullius in verba” - “take nobody’s word for it.”

If you’re ready to start your move to social intelligence, get inspired at data.twitter.com.

¹ ESOMAR Global Market Research Study 2020