Back to
Digging for gold in your own data

Digging for gold in your own data

Published on 07/12/2013

There are rich seams of gold to be mined from the vast troves of data residing and flowing through every business.

However, the difference between striking a mother lode and giving up in despair lies in the attitude of a firm’s leaders.  If decisions are made in an ad hoc “seat of the pants” fashion or cultural barriers such as “we have always done it this way” exist, gains from analytics will be desultory.

If a firm’s leaders are curious about what is happening, why it is happening, what may happen and, most importantly, what should be done about it, just digging on the surface of the firm’s data deposits will likely produce a high yield from analytics.

Where to start?  Simply, follow the money.  What products (SKUs), services, channels or customers have the greatest potential to impact on shareholder value?  Academic studies have shown that up to 40% of a firm’s customers, products and transactions are unprofitable, so it will not be difficult to find a target.  Select a small and high potential segment to ensure early successes are valued and lead to a greater investment in analytics.

The quality and availability of a firm’s data may also strongly influence which projects are selected.  All existing firms have legacy systems and silos of data, which are often not easily accessed.  A critical initial step is to organise the relevant data into a single source of truth, which, when accepted by the firm’s leadership, will avoid unproductive debates about the veracity of the source data.

The essential value of business analytics is to help explain cause-and-effect relationships.  Determining which analytics to employ depends on understanding how all of the moving parts work together.

To accomplish this, analysts will gain enormously by being embedded within the relevant business unit, rubbing shoulders with line managers who deal daily with subtle complexities and nuances of the business.

The ability to compete on analytics depends on cultural adoption, as well as the maturity level of analytics adoption.  Almost all firms have some existing level of analytics adoption.

The hierarchy of competitive advantage from analytics may be split into descriptive analytics and predictive or prescriptive analytics.  Descriptive analytics begins with basic standard reports, which explain, “what happened”, includes query capability to determine the problem, scorecards to describe what information really matters and concludes with alerts, which may require action.

Predictive and prescriptive analytics, by contrast, begins with statistical models, which help to explain causes and effects, includes randomised testing to consider what might happen with a new approach, and predictive modelling, which helps to consider what may happen next. At this stage of the hierarchy, a firm may be described as an “analytical company”.

The final hierarchical stage of analytical capability, or Holy Grail, is optimisation, which helps to determine the best that can happen.

Optimisation requires a firm’s resources and opportunities to be modelled effectively. Naturally, any model is an abstract of reality and can never fully model all of the real world complexities that impact on a firm.

One example of optimisation is to select investments from a range of options.  An optimisation algorithm, or mathematical process, calculates the best choices to maximise the value of the firm. This is known as a deterministic solution, which is a single outcome for a specific set of assumptions.  It does not take into account risk.

Predictive analytics helps to provide an understanding of the probability of possible future outcomes, based on past trends.  This may be validated and supplemented by the business leaders’ own knowledge of future likely outcomes of key variables.

The gold-standard analytical tool to address future uncertainty is Monte Carlo simulation. Probability distributions for key variable are combined to generate values to feed into the optimisation model.  With enough scenarios run, an overall probability distribution of optimised firm values is generated.  In this case, the optimal investment choices are determined while explicitly considering future uncertainty, given the limitations of the modelling.

The calculations required in a realistic situation are computationally intensive. Cloud computing is a saviour in this regard.  Literally thousands of servers may be orchestrated to run through tens of thousands of risk and optimisation scenarios for just hundreds of dollars.

Wherever the firm is on the analytical adoption curve, significant benefits will accrue.  However, the critical success factor determining how far a firm goes along the adoption curve towards becoming an analytical competitor is how passionate and committed the firm’s leadership is to reaching that goal.

– Dr Michael Snowden, Managing Director


Download the PDF, as featured in The National Business Review, 12 July 2013