14 Aug 2020

Manage Data Overload & Improve Decision-Making

Define the questions that need to be answered to reduce the chance of data overload and improve decision making.

The side effect of working with big data can be dangerous: it’s called data overload, and it can overwhelm newbies and experienced people alike.

Data these days is big. More than 2.9 billion emails are sent every second, according to Internet Live Stats. Data specialist Bernard Marr says we create as much data every two days as we did from the beginning of time until 2003. Yep. Data overload is real.

The art of working with data is more about transforming it into decisions and insights than dealing with the coding, algorithms or databases. All the math's and statistics mean nothing if you can’t use big data to create business advantage.

Avoiding getting lost is the key. Creating your own map by asking the right questions of data will help you navigate the vast and choppy waters of data overload.

Data science lecturer and author of From Business Intelligence to Data Science Alex Scriven says data overload happens for two reasons:

  1. The business hasn’t defined the questions it wants answered
  2. The data product creator has simply thrown everything in because they have it

The Centre For The Future founder Dr Richard Hames - who uses a proprietary AI algorithm to ingest public data and make predictions about trends ahead - says many businesses and governments may not have sufficient strategic insight to make good decisions from data.

“There’s a distinction between complexity - and seeing the pattern - and what is complicated,” Dr Hames says.

“Most businesses and governments are stuck in complicated legacy systems rather than embracing the complexity to make better decisions.” Hames urges businesses to think outside the box to ask the right questions.

How to avoid data overload

CSIRO Data 61 principal researcher Dr Surya Nepal says not all data is important. Businesses need to find what matters - what is core to their business - and work with that.

It’s best to start with the business questions that need answering. Before jumping into data analysis, it’s better to create a dashboard, report or presentation to answer one to three key questions.

Blue Bricks CEO Vikram Sareen says working with data lakes provides massive advantages - but you don’t want to drown in them. There are four simple steps to mitigate being overwhelmed by data:

START WITH THE END IN MIND: All good science starts with a hypothesis, as does working with data. Know what you are trying to measure, predict or analyze and define the problem or questions upfront.

MINIMIZE DATA COLLECTION: Don’t collect or process data that you don’t need.

CHECK THE BEST DATA FORMAT: Data analysis is tricky - just ask anyone who has had to wrangle columns into rows or clean up time series data. Make sure you collect data in the format and structure that’s best for analysis.

VISUALIZE THE ANSWERS TO THE QUESTIONS: No-one wants to read thousands of rows in a spreadsheet or read walls of text describing the insights. Best practice is to use dashboards and visualizations to communicate the answers to the questions you started with in step one.

What about metrics and measuring everything?

Finding the right metrics, KPIs or OKRs to track is a business decision rather than a data decision.

Useful statistics - and metrics - tend to reveal the outcome of an action at one time will be similar to the outcome of the same action at a later time. They are ideally specific and measurable at a point in time. When metrics are really good, they can be predictive and demonstrate causal relationships. 

Choosing the right things to measure allows a business to track and manage the cause-and-effect relationships that determine value. Choosing the wrong things wastes a whole lot of time and resources.

Drivers of business value - and the questions you want to ask - may change over time in which case so will your data collection and reporting. Then it becomes important to make sure you compare apples with apples and not mix up the results by collecting ever-changing data. Always optimizing your data collection and analysis is no-brainer.

“We need to understand the life cycle of collecting data, preparing it, its fitness for certain algorithms and so on,” says NSW Government data scientist Ian Oppermann.

Big data lets you ask bigger questions

While measuring metrics every month can be valuable, the real art of embracing what’s possible comes when using big data - and merging different data sets - to ask almost unknowable questions.

“Data science is great for solving problems - it can take large volumes of data and find patterns or build models to predict things,” Oppermann says.

Building models to forecast trends, optimize pricing or reduce customer churn is now commonplace in businesses. But it takes skilled data professionals - and perhaps an engineer or data scientist – to do it well.

“The warehousing, cleansing and anonymisation of data is something that engineers do well, but retailers or business people want to ask questions of the data - for example, ‘What sales will we do at Christmas?’,” explains Blue Bricks CEO Vikram Sareen.

The real art is finding the right question to ask.

Looking to upskill in data driven marketing? Here are our recommended courses from ADMA IQ:

Need more info?