Will Big Data And Analytics Deliver The Promised Land?

This post got published before I intended to publish it. Sorry for this oversight. I have now completed it as intended and am republishing it. I apologise for any inconvenience and thank you for your understanding.

What do B2B technology vendors sell?

No, it is not the technology.  Think again, what do B2B technology vendors sell?  They sell dreams that speak to a fundamental human need. What dreams? Dreams of control-mastery-domination over the ever flowing, every morphing, character of a process we turn into a noun: life.

What need do these dreams take root from and speak to?  The need for safety and security. At some fundamental level we get that nature is indifferent to our survival and wellbeing.  To deal with this anxiety we embrace anything that provides the illusion of safety-security. The Greeks embraced the Gods, we embrace technology and the latest technofix.

I notice that the big data and analytics space is hot right now.  It is the latest technofix being pushed by the B2B technology vendors.  It occurs to me that this technofix is designed to speak to those running large enterprises – especially those who are higher up and divorced from the lived experience of daily operational life at the coal face.

What I find astonishing is that so few actually ask the following two questions:

1. “What kind of a being is a human being?”

2. “What kind of a culture is human culture?”

What is the defining characteristic of human beings?

Allow me to illustrate by share a story I read many years ago:

Psychologist:  John, you have been referred to me by the authorities. They tell me that you think that are dead. Is that right? Are you dead?

John: Absolutely, I died a little while back.  I am dead. 

Psychologist: How interesting! You died a little back. Yet here you are talking with me. And I am not dead.  So how is it that you are dead and I am not dead, yet here we are talking? 

John: Beats me how this works or why it is happening. I know that I am dead. 

Psychologist: John, I have an idea. Do dead people bleed? 

John: Don’t be ridiculous! Everyone knows that dead people don’t bleed! 

The psychologist suddenly reaches over and cuts John’s hand with a knife. Both of them are looking at John’s hand. Blood, dark red blood, is seeping through the cut.  The psychologist looks at John with the look of satisfaction, of victory. Let’s rejoin the conversation.

Psychologist: John, do you see that blood on your hand? How do you make sense of it? You say that you are dead. And earlier you told me that dead people don’t bleed.

John: F**k me, dead people do bleed!

This is not simply an amusing story.  It is a story that captures the experience of a respected psychologist who has been dealing with many kinds of people, dealing with many kinds of problems, over a lifetime.  This story capture a fundamental truth of the human condition.

It appears that to survive in the world as it is and as we have made it, we need to be deluded. We need to distort reality: to make life more predictable, to make our current situation lighter-better than it is, to see a future brighter than is merited by the facts, to see ourselves stronger, more capable, more influential than we are. Studies suggest that those of us who lack this ability to distort reality and delude ourselves end up depressing ourselves.

What Kind Of A Culture Is Human Culture?

Symbolic and ideological.  Why?  Because human beings just don’t cope well with the world as it is. So we get together into tribes. And the glue that keeps the tribe together is a particular way of constructing the world, a particular way of giving meaning to the world, and a particular way of interacting with the world.  And when I speak world I include human being, and human beings; a human being is always a being-in-the-world as in always and forever an intrinsic thread in that which we call world.

The next question: which ideology do members of society espouse?  The dominant public ideology. In the world of business this is that of scientific management and in particular reasoning and making decisions objectively – irrespective of the past, of tradition, of our personal interests and opinions.

A more interesting question is that about the actual behaviour of the elites, the Tops. What is it that the Tops actually do?  They do that which protects and furthers their interests: their power, their status, their privileges, their wealth, their dominance.  So insight and recommendations (whether from big data and analytics or through conventional methods) that are in line with these interests are heartily accepted and actioned swiftly and vigorously.

Any insights and recommendations that challenge the vested interests of the elite (Tops) are repressed at the individual level, belittled-disputed-ignored at the societal level.  I invite you to read this article which can be summed up as the UK Government sacks the chair of the official Advisory Council on the Misuse of Drugs. Why? Because the chair was insisting on the reclassification of drugs. What happened?

  1. The Advisory Council looked at the data (of harm to the individual taking the drugs and others affected by his/her behaviour) on drugs at the request of the UK.

  2. On the basis of the data, the Advisory Council came up with the conclusion that “if drugs were classified on the basis of the harm they do, alcohol would be class A, alongside heroin and crack cocaine.”

  3. The drug rankings, associated findings and recommendations were ignored by the UK government. Why? Because they went against the government’s stance on drugs.

  4. The chair of the Advisory Council challenged the UK government’s refusal to act on the recommendations of the Advisory Council.  So the appropriate UK Government minister sacked him.

What Does The Future Hold for Big Data & Analytics?

If past behaviour is an adequate guide to the future then it is safe to say that technology vendors will get rich. And the business folks will have another layer of technology that they have to manage. One or two organisations may reap substantial benefits, the rest will be disappointed.  Yet, this disappointment will not last long. Why? By that time the technology folks will have come up with the latest technofix!

I leave you with the following thoughts:

1. There are no technofixes to the kinds of social issues-problems we continue to face;

2. Incremental improvements lie in the domain of big data and analytics;

3. Breakthroughs lie in our ability to see that which is with new eyes – a shift in dominant concepts, dominant paradigm, dominant ideology, dominant way of seeing that which is.

Put differently, big data & analytics is a red herring for those who aspire to lead: to cause-create that which does not exist today.  Managers, those whose horizon extends to daily operations and the next twelve months, may find big data and analytics useful – as long as it does not threaten the sacred cows of the Tops-Middles and the corporate culture.

Want to get value out of your data and analytics investment? Then deal with this issue before you buy the software.

‘Rationals’, data and the wonders of analytics

I think I have said this before and I will say it again: my fascination is with us – human beings being human beings.  In particular I am fascinated by ‘Rationals’.  Whom I am speaking about when I speak ‘Rationals’?  I am pointing at/towards my fellow human beings who pride themselves in being ‘rational’, ‘objective’, ‘scientific’ – they usually love order, logic, reason and are attracted to / come from ‘engineering-science-accountancy-economics-mathemetics” type disciplines.

What is it that I find fascinating about my fellow ‘Rationals’?  Before I go further I should point out that I used to be a ‘Rational’ and do get sucked back into being a ‘Rational’ if I am not being mindful.  What do you expect?  I studied Mathematics, Physics, Chemistry.  I have a BSc in Applied Physics and then went on to study accountancy and qualified as a Chartered Accountant.  OK back to my question: what do I find fascinating about ‘Rationals’?  Bluntly speaking, ‘Rationals’ are the most irrational people and they totally do not get this paradox, this joke.   Put differently, ‘Rationals’ are blind they are to the way that human beings work, organisations work, society works.  They are suckers for universal principles and insist that the world act in accordance with these universal principles.  One of these fundamental “shoulds” is that “people should behave rationally”.  Why does this matter?

Right now there is a fad in progress.  The people behind this fad hail, loudly and frequently, the numerous wonders (and benefits) of what data and analytics can do.  I suspect that many of them are ‘Rationals’ with a good sprinkling of ‘Marketing’ types thrown in.  They proclaim that big data and state of the art analytics (social, content, text, predictive…..) will light up our world and lead to the promised land: mountains of revenue; costs trimmed to the bone – everything working so efficiently so as to render void the 2nd law of thermodynamics (order to disorder) and the messiness of the real world; and an ocean of profits.  And all you have to do is to mine the Big Data!

The ideal world: how ‘Rationals’ assume the world works

The ‘Rationals’ assume that every person and certain every influential person making decisions is John Maynard Keynes (the famous economist).  What do I mean?  He was once asked how he responds to new data that did not support his earlier decisions and judgements.  JMK replied “I change my opinion. What do you do sir?”

Yes, that is the ideal.  Every little one of us as a perfect computer: taking in data as input, crunching that data against any number of dependable algorithms, spitting out the answer and doing what is in line with that answer ignoring our ‘points of view’, our ‘prejudices and bias’.  Now lets take a look at reality.

The real world: human beings are strange, marvellous, creatures

Daniel Kahneman in his latest book (Thinking, fast and slow) spells out that human beings are essentially a meaning making organism that thinks/works in stories and jumps to instant conclusions as long as the story fits the preconceived schema.  He writes “The implication is clear: as the psychologist Jonathan Haidt said in another context, “The emotional tails wags the rational dog.” The affect heuristic simplifies our lives by creating a world that is much tidier than reality….” Mr Kahneman has titled one his chapters “Causes trump statistics“.

Lets just ignore the fact that the people interpreting data and presenting it to decision makers are human and they exhibit the same ‘marvels and failings’ as the ordinary person: in test after test conducted by Daniel Kahneman and Amos Tversky the professional statisticians made similar ‘errors’ to the ordinary person.  By ignoring this you would think I have just spelled out an excellent reason to deploy analytics to drive decision making – to take out these human failings.  If only the world was that simple my ‘Rational’ friend.  Let me share three stories with you to illuminate the nature of the real world.

Story 1.  On page 116, Mr Kahneman tells the story of how Tom Gilovich and Robert Vallone did a statistical analysis of thousands of sequences of shots in basketball and concluded that there is no such thing as a hot hand in basketball.  Thats right according to the data and the statistical interpretation of data, there is no such thing as a basketball player having a hot hand – it is a human invention.  Now here is the instructive part, how did the basketball public (coaches, media, fans…) react to this conclusion?  Disbelief – just in case you don’t get that it means they did not believe it!  This is what Red Auerback, the celebrated coach of the Boston Cetics said: “Who is this guy? So he makes a study. I couldn’t care less.”

Story 2.  There is an amusing and enlightening story about human beings and it goes like this.  Once upon a time an ordinary villager died.  His body was cleansed, taken to the local mosque, the villagers gathered together at the mosque, the prayers were said and all the necessary rituals performed with the family present.  Then the body was put into the coffin.  Some of the villagers lifted the coffin on their shoulders and headed for the cemetry.  On the way they heard a knocking coming from the inside of the coffin. Then they heard a voice say “Let me out of here. Let me out of here. I’m alive, why have you put me in this box?  Let me out of here!”  The folks carrying the coffin replied “No, this is a trick.  You are dead.  We have said prayers and carried out all the necessary rituals.  Now it is time to bury you.”  No matter how much the villager pleaded to be let out the folks carrying his coffin refused to listen.  They were adamant that he could not be alive after all hadn’t all the villager seen his dead body, said prayers….. they would not be made fools of by the ‘dead man’ and end up being laughed at by the villagers when they returned to the village, opened the coffin and found that the the voice coming from the coffin was a hoax!  No, he was dead, everyone knew that!

Story 3. During World War II there was a notable disaster – Operation Market Garden.  This is what Wikipedia says, “Montgomery was able to persuade Eisenhower to adopt his strategy of a single thrust to the Ruhr with Operation Market Garden in September 1944….. the operation failed with the destruction of the British 1st Airborne Division at the Battle of Arnhem and the loss of any hopes of invading Germany by the end of 1944.”  What makes this relevant is the fact that this disaster could have been avoided.  How?  Brian Urquhart.  This is what Wikipedia says, read it carefully:

“In the autumn, as the 1st Airborne Corps Intelligence Officer, he assisted with the planning for Operation Market Garden, an ambitious airborne operation designed to seize the Dutch bridges over the rivers barring the Allied advance into northern Germany. He became convinced that the plan was critically flawed, and attempted to persuade his superiors to modify or abort their plans in light of crucial information obtained from aerial reconnaissance and the Dutch resistance. The episode was described by Cornelius Ryan in his book on “Market Garden”, A Bridge Too Far. (In the film version, directed by Richard Attenborough, Urquhart’s character was renamed “Major Fuller”, to avoid confusion with a similarly named British General.) ……… but he became deeply depressed by his failure to persuade his superiors to halt the operation and requested a transfer out of the airborne forces.

General Browning ignored the intelligence supplied by Brian Urquhart, the intelligence officer.  Why?  He didn’t want to have to tell his boss Field Marshall Montgomery that he was cancelling another operation – many airborne operations had previously been cancelled.  How did the operation turn out?

So what do we have here?  We have an intelligence officer doing his job and providing the intelligence.  The intelligence goes against what the top brass is committed to and the intelligence is ignored, the operation goes ahead and it is a disaster.  Do you think this is a one-off event?  It happens all the time: think Iraq and WMDs; think of the financial crisis and the reckless lending that led to it and the warnings that were ignored….  Does the Sufi story (told earlier) sound far fetched now?

If you want to get value out of data and analytics then deal with human nature as it is

I love zen – it says see life as it is and as it is not, leave behind your mind full of theories, concepts, projections of how you want things to be.   You would do well to act on that advice before you spend a fortune on “big data and analytics”.   Why?  It is not easy to get the right data (information overload as much of an issue as data quality and data integration) and convert that into useful intelligence that can drive decision-making.  No it is not easy.  Yes, I do know that the smooth tongued marketers are promising you that it is so easy.  The reality is that it takes time (lots of it), effort (lots of it) and money (lots of it).  Yes, the technical aspect is doable if you put in the time, effort and money that is required: the experts will come in and do it for you.

The hard part, the really hard part, is the human part.  It is dealing with, working with, human nature as it is. Human beings do not get data and do not value data – they simply do not relate to it like they relate to, say, a dog.  Do you really think that the people in your organisation appreciate the value of data and put in the time and effort to enter the right data into the right fields?  If you think that then you have spent far too much of your time in the Ivory Tower of the executive office.   Yet even that issue – of getting your people to enter the right data into the right fields/systems – is insignificant to the real issue.  What is the real issue?  Getting managers to give up their pet theories, their ideological convictions, their vested interests, their intuition, their past experience and use data and analytics to make decisions.  That is the central issue that you have to and should deal with.

My advice

Before you go and spend a fortune on ‘big data and analytics’ do the following:

Find out the total cost of ownership.  The set-up cost (people, technology, other) and the on going operational cost.  Statisticians don’t come cheap and then their are annual software licensing fees to think of…

Hold a ‘Big Data and Analytics’ party and invite all the key people from you business to that party.  Spell out the wonders that ‘Big Data and Analytics’ will bring them and the company.  Then ask them to pay (out of their existing budgets as they are) to attend the party.  The price of admission to this party?  Just divide the total cost of ownership (say over three years) between each of the players in the organisation.  Then see who turns up.  That might give a true picture of how much passion there is for ‘Big Data and Analytics’ within your organisation.  Or you can try the “build it and they will come” approach – your party, your choice!

Why big data and analytics will not make you the next Apple

Are  ‘big data’ and ‘analytics’ the latest fads?

In my view there is a little too much hype around ‘big data’ and the power of analytics to drive business growth and profitability.  Can ‘big data’ and ‘analytics’ help you improve operations?  Yes – they can help you better staff and manage your call centre  or improve your supply chain or target your marketing better.   Will ‘big data’ and analytics make you the next Apple?  Highly unlikely.  Why does that matter?  Because whilst you are busy optimising operations someone else is inventing a future in which your optimised operations become redundant: think music, think publishing, think mobile phones.  Let’s explore this further.

What are the root causes?

Years ago when I worked in the business planning & analysis team of a brand name drinks company I noticed something interesting.   Managers could drill down and find out which particular markets had failed to make their numbers.  Great – we know which business unit is underperforming.  But why is this unit doing better or worse than expected?  This is where the fun started.  First, there were almost as many opinions as the people we asked.  Second, there was no easy to work out whether the answers given by local management held any resemblance to the ‘truth’.  When I dug further I found out that local management did not know why they had failed to make their numbers (revenues).  When they were asked, the local managers simply came up with the most plausible story.  So month end because a ritual where HQ asked questions and the local managers invented plausible stories.

Lesson 1:  For any complex event there are a multitude of plausible answers and working out the ‘real reason’ is notoriously difficult.

Lesson 2:  Without a sound grasp of the root causes it is difficult to formulate a sensible course of action to address the situation.

What do we do about it?

Lets assume that you have all the data and analysis has been done.  You know there is a problem.  What do you do about it?  Is it easy to get all the actors – who have to play ball – to agree on the course of action to take?  In my experience the answer is no: the more ‘strategic’ the issue at hand the more difficult it is to come to an agreement on a sensible course of action.  Let’s take a look at the recent banking crisis.  Did all the main actors (the western economies) come to a consensus on what course of action to take?  No.  Just take a look at the Euro crisis: why is it that the leaders of the EU cannot agree on the right course of action?  First, because different people have different ideas about what constitutes the right course of action.  Second, the right course of action from an objective perspective may simply not be viable from a political perspective.

Lesson 3: The more that is at stake the harder it is to get all the actors to agree to a single course of action and then act to play their part and execute that course of action

Lesson 4: If you are unable to act decisively and as a single unit then all the data, analysis and insight is worthless

Does it really tell you what you need to know to thrive in the future?

Larry Freed has written an article that resonates with me.  He points out that you need to be clear on what you know and what you do not know.  I’d say that you need to be clear about what data and analytics is telling you and what it is not telling you.  Larry, talking about a website, asks do you know:

  • Why visitors come to your site (to research, to buy, to complete a transaction, to get product support, to learn more about your company before interacting with you through another touch point, etc.)?
  • What influences visits to your site (a referral, a social media interaction, a failure to resolve an issue with a call centre, an advertisement, a news story, a previous affinity with your brand, etc.) and which customer acquisition sources result in traffic that is the mostly likely to convert?
  • What visitors need from your website? How needs differ by population segment or other segmentation that is useful to your business—perhaps first-time vs. repeat visitors, heavy users vs. light users, etc.?
  • What visitors expect from your website? Do men and women have the same expectations? Old and young? Do people who arrived as a result of a Google ad have the same expectations as those who arrived because of a TV ad?
  • What channel your visitors prefer, and are there ways you can influence that preference so they frequent less costly, more profitable channels?
  • How customers view your business, compared to the way non-customers view your business, relative to your competition?
  • How your customer profiles and expectations change in response to market and broader economic conditions? And what, if anything, you need to change as a result?”

Colin Shaw in a recent post makes the same point in a different way.  Here are some relevent extracts from his post:

“Google and Facebook, as Eli Pariser discusses as a part of TED, are engaged in the process of quantifying preferences from the timing and frequency of online clicks, and using this information to alter web content. The stated goal of this practice is to “personalize” the web experience.

A lesson gleaned from Dell’s 1990s laptop boom illustrates the point that preference and value are two different things. Dell let its customers customize all aspects of their computer’s hardware – from screen size to keyboards to RAM – everything but color. Nobody thought to customize color, because a laptop was supposed to be black or gray. However, when color laptops were introduced, sales skyrocketed and we all learned that color was indeed an important factor.

Imagine if Google and Facebook had monitored “clicks.” They would infer that because customers did not indicate a laptop color preference, it doesn’t drive value and is therefore irrelevant.”

Lesson 5: there is an assumption behind all predictive analytics and that is “all things being equal” – that is to say that predictive analytics assumes that the future will be a replay of the past.

Lesson 6: human behaviour is shaped by the ‘structures’ in which human beings are embedded, change the structure and you are likely to see human beings change their behaviour. Think about how the recession (e.g. job losses) have changed the shopping habits of consumers in the western economies.

Why won’t big data and analytics make you into the next Apple?

Apple was busy creating a new future (a break from the past) rather than exploiting the past.  If you take a look at the US automotive industry the big US automakers were busy building and selling gas guzzlers because the analytics showed that these were the cars that Americans were buying.  At the same time Toyota was busy living into a very different vision of the future: hybrid cars and electric cars.   Who was right?  According to the data and the analytics it was the US automakers.  What would you say now?  Toyota?

If you are not inventing the future you can still prosper by picking up the weak signals that point towards a new trend.  I once asked Bob Greenberg (R/GA) the secret of his success and he told me it was his ability to see these trends and act upon them before others.  You might imagine that analytics might help you to spot trends.  My experience of traditional analytics is that the modelers do all they can to strip out the outliers and create a normal distribution so that the maths works – in doing that they filter out the ‘weak signals’ that point towards these trends.

Perhaps it is best to end by remembering what Colin Shaw points out: how would analytics have disclosed that customers wanted to customise the colour of their laptops and that once this option was made available then Dell’s laptop sales would surge.

What do you think?  If I have it wrong then please do educate me.