Why bigger data sets don’t mean better insights
‘Data is the new oil’ is a term coined in 2006 by British mathematician Clive Humby. It has become a widely used term that mainly means that if your organization has access to large amounts of data, you can use it to support decision making and drive results.
While there is great truth in the fact that access to data can lead to greater business intelligence insights, what companies actually need is access to “good” data and the insights within it. However, knowing what makes data valuable is something that many still struggle with. Because considerations often include factors like quantity, age, source, or variety, not truly understanding what type of data is good for the business means it’s easy to get lost in data sets that are ultimately of poor quality and bad for decision-making.
The major costs of the wrong big data
The costs of processing poor quality data are high. On average, the price is $13 million per company or 10-30% of revenue, and for companies of all sizes that is a huge burden.
Companies have become accustomed to making decisions based on big data sets. They use spreadsheet software to analyze and use that analysis to make decisions. But this approach fuels the need for even more data that can identify trends of ‘statistical significance’. The challenge is that it is difficult to really scrutinize the source and authenticity of information. Take consumer insights, for example. If a company obtains its data sets from a third party, can it ever be 100% sure that all that information was provided by authentic respondents, and that nothing from bots or humans is not 100% truthful?
Co-founder and CXO, GetWhy.
For case studies on why large data sets don’t always produce more accurate results, we can look to politics. 2024 will be a monumental year, with elections taking place in both the US and Britain and political opinion polls once again playing a role in predicting outcomes. However, they are not always right. During both the 2016 and 2020 US elections, pre-voting polls incorrectly predicted some very big things; the first even predicted that Hilary Clinton would celebrate a big victory. Reasons for the wildly incorrect predictions include non-response bias, where Trump voters were less likely to interact with pollsters, skewing the results toward Clinton.
Similarly, in Britain, the ‘Shy Tory factor’ has been referred to in elections where the Conservative Party has performed better than the polls predicted. In this case, respondents would say they would vote in the opposite direction of what they would ultimately do.
While a handful of such respondents in large data sets may not have too much influence on the final analysis, the aforementioned election polls show what can happen when the data is not truly reflective of the outside world. For companies that use such analytics to guide decision-making, acting on that information can come at a high cost.
Listening versus understanding
Relying on big data sets is also a sign that companies are often geared to listen to, not understand, their consumers. This means that while they can use big data to see trends, they don’t understand why those trends exist. For example, if an organization knows that consumers like the color blue, but does not seek further information, it has simply listened. This may prove successful in the short term, but if that trend suddenly changes and consumers start liking green, they will be slow to respond.
If a company knows that consumers like blue, but goes a step further and discovers why they do, it will understand what actually influences them. Perhaps blue is a response to an event or a certain mood, and having that information allows an organization to not only make decisions that are more empathetic to consumers, but also better prepare for any evolution in requirements.
Ushering in a new era of empathy
Empathy is crucial at a time when the world is facing challenging times. With several major geopolitical events taking place, understanding consumers is one of the many things that can help usher in a new era of empathy. Companies also need to do work to keep consumers on side as there is a growing distrust of brands caused by a number of reasons. For example, consumers are often exposed to unfair practices online, including fake reviews and data issues surrounding targeted advertising.
To break the cycle, companies must rethink how they discover insights. Gathering insights has typically involved enormous time and cost investments, and the resulting large cumbersome data sets that dehumanize respondents are no longer appropriate in a world where people’s views are constantly changing. Not only does it take too long to collect, but the data may be incorrect in the first place.
Organizations need to place more emphasis on understanding consumers. They need to know why they think a certain way, not just that they do. AI-driven qualitative insights allow companies to quickly understand what audiences really want. The AI can run survey tools with respondents from demographic groups around the world before delivering analysis within hours, and with the same quality as traditional methods. By subsequently watching the recordings, brands see not only what the respondent says, but also how and why they say it.
Ultimately, bad data costs companies a lot. Acting on information that is not inaccurate can have significant consequences, ranging from slightly dissatisfied consumers to complete failure. Companies need to ditch their old processes and adopt a new approach to gathering insights. Larger data sets don’t mean better insights; a more thoughtful, targeted approach is. And when companies truly understand consumers, it drives empathetic decision-making, brand trust and better results.
We have listed the best customer database software.
This article was produced as part of Ny BreakingPro’s Expert Insights channel, where we profile the best and brightest minds in today’s technology industry. The views expressed here are those of the author and are not necessarily those of Ny BreakingPro or Future plc. If you are interested in contributing, you can read more here: https://www.techradar.com/news/submit-your-story-to-techradar-pro