Reason blog post

The problem statement

Disappointment seems to be the view of the majority of C-level executives. It appears that despite the ever increasing maturity of technology, the full potential of big data is still not being realised.

As is the case with many a buzzy term, we’ve been blinded by the promise and forgotten about the practicalities.

Doug Laney popularised the term 'Big Data' back in 2001 and since then it has become one of the primary areas of tech investment for Fortune 500 companies, with IDC estimating $150bn being spent in 2017 alone.

But, as with so many technology revolutions, the amount of real revolution achieved has been limited to some outstanding achievements from a few large players. The majority of organisations have seen negligible return on the considerable investment they have put in.

So is the party over? Not necessarily, many of the original promises of Big Data remain true, and the high profile failures (US election predictions? Brexit?) are not failures of the underlying concepts but more failures of implementation.

"What is needed is a more pragmatic approach to realising data benefits, through shorter implementation cycles and greater iteration."

~ David White - Data Strategist at Reason

Creating an environment for successful data experiments

Before we dive into solutions, let’s look at why an estimated 73% of Big Data projects fail.

Having been involved in many data projects, we believe certain conditions must be met for successful outcomes:

A clear hypothesis or hypotheses
A common error is to not understand the questions you are hoping to get answers to.

Accessible and recent data
Work with the data you have in the short term, building a quality data landscape takes time.

Insight is pointless unless it unearths opportunities for commercial innovation.

Further data must be gathered as part of a feedback loop.

Hypothesis refinement
Feedback should lead to better questions being asked of the data, and the process recommences.

If you are aware of the various flavours of Lean methodology, including Eric Reis and the Lean Startup movement, then this process will be familiar. These methods are tried and tested and have worked at companies from Google to Toyota.

"Taking the key principles of a Lean approach to product development leads to better outcomes when applied to data projects."

~ David White - Data Strategist at Reason

Lean Data

In an industry often crippled by jargon, we’re - cautiously - referring to a Lean Data approach. Let’s take a look at how this plays out in reality:

Minimise waste
If you only take one bit of advice from this post it should be 'don’t use more data than you need’.

Just because you’re doing ‘Big Data’ doesn’t mean you have to use all the data, this is a frequent mistake and burns time and resource with a high risk of failure. The noise will more often than not drown the signal. 

Create a data landscape that presents that smallest relevant data set to build your model, and only add more data through improvement loops (iterations). View this as your Minimal Viable Product.

Continous improvement
Gather data on the impact of your changes and iterate, iterate, iterate! Nobody does a perfect version one, the process of iteration is where the game is won or lost.

Measure against the big picture
Success of any data project should be measured against both tactical and strategic goals.

Insight is all very well but unless it moves the needle it is pointless. If your insights and resulting changes are not moving the needle, stop. Using a Lean Data approach means that the experiments will be a low cost and hence stopping should be a natural part of the process.

Data presents a significant opportunity for the majority of businesses, but the scale of the upside is matched by the level of risk. Lean methodologies allow for that risk to be managed with a fast-to-product, fast-to-fail, fast-to-learn mindset.

In short: using Lean Data principles will result in a better return on your data investment.

To learn more about ‘Lean Data’ and how it can help with your challenges, drop us a note here or fill in the form below.

Get in touch to discuss

Please enter your name
Please enter your comments
Please enter a valid email address