Musings on Big Data, Customer Analytics, and Data Driven Business

On LinkedIn, Don Peppers is sharing his perspective on making better decisions with data.  This got me thinking and I want to share with you what showed up for me. Why listen to my speaking?  I do have a scientific background (BSc Applied Physics).  I qualified as a chartered accountant and was involved in producing all kinds of reports for managers and saw what they did or did not do with them. More recently, I was the head of a data mining and predictive analytics practice. Let’s start.

Data and data driven decision-making tools are not enough

Yes, there is a data deluge, and this deluge is becoming down faster and faster. Big enough and fast enough to be given the catchy name Big Data.  What is forgotten is the effort that it takes to get this data fit for the purpose of modelling.  This is no easy-cheap task. Yet, it can be done if you throw enough resources at it.

Yes, there are all kinds of tools for finding patterns in this data. And in the hands of the right people (statistically trained-minded, business savvy) these tools can be used to turn data into valuable (actionable) insight.  This is not as easy as it sounds. Why?  Because there is  shortage of these statistically trained and minded people: amateurs will not do, experts are necessary to distinguish between gold and fools gold – given enough data you can find just about any pattern.  It statistical savvy is not enough you have to couple it with business savvy. Nonetheless, let’s assume that we can overcome this constraint.

The real challenge in generating data driven decision-making in businesses is the cultural practices.  We do not have the cultural practices that create the space for data driven decision-making to show up and flourish.  A thinker much smarter-wiser than me has already shared his wisdom, I invite you to listen:

On the whole, scientific methods are at least as important as any other research: for it is upon the insight into the method that the scientific spirit depends: and if these methods are lost, then all the results of science could not prevent a renewed triumph of superstition and nonsense.

Clever people may learn as much as they wish of the results of science – still one will always notice in their conversation, and especially in their hypotheses, that they lack the scientific spirit; they do not have the distinctive mistrust of the aberrations of thought which through long training are deeply rooted in the soul of every scientific person.  They are content to find any hypothesis at all concerning some matter; then they are all fire and for it and think that is enough …….. If something is unexplained, the grow hot over the first notion that comes into their heads and looks like an explanation ….

– Nietzsche (Human, All Too Human)

It occurs to me that the scientific method never took route in organisational life. Put aside the rationalist ideology and take a good look at what goes in business including how decisions are made. I say you will find that Nietzsche penetrating insight into the human condition as true today as when he spoke it. The practice of making decisions in every organisation that I have ever come in contact with is not scientific: it does not follow the scientific method. On the contrary, managers make decisions that are in alignment with their intuition, their prejudices, and their self-interest.  It is so rare to come across a manager (and organisation) that makes decisions using the scientific method that when this does occur I am stopped in my tracks. It is the same kind of unexpectedness as seeing a female streaker running across the football pitch in a league match.

What are the challenges in putting data driven decision-making practices into place in organisations?

Technologists have a gift. What gift? The gift of not understanding, deeply enough, the being of human beings. Lacking this understanding they can and do (confidently) stand up and preach the virtues-benefits of technology.  If life were that simple.

Truth shows up as attractive to those of us who do not have to face the consequences of truth.  Data driven decision-making sounds great for those of us selling (making a living and hoping to get rich) data driven tools and services.

The challenge of putting in place data driven decision-making practices is that it disturbs the status quo. When you disturb the status quo you go up against the powerful who benefit from that status quo.  Remember Socrates:

The very nature of what Socrates did made him a disruptive and subversive influence. He was teaching people to question everything, and he was exposing the ignorance of individuals in power and authority. He became much loved but also much hated …. In the end the authorities arrested him for …., and not believing in the gods of the city. He was tried and condemned to die …

– Bryan Magee, Professor

Beware of being successful in putting in place a culture of data driven decision making!

With sufficient commitment and investment you can put in place a data driven decision making culture. Like the folks at Tesco did.  And by making decisions through harnessing the data on your customers, your stores, your products, you can outdo all of your competitors, grow like crazy and make bumper profits.  Again, again, and again.  Then the day of reckoning comes – when you come face to face with the flaws of making decisions solely on the basis of data.

Tesco is not doing so great.  It has not been doing so great for several years – including issuing its first ever profits alert in 2012.  What is the latest situation?  Tesco has reported a 23.5% drop in profits in the first half of this year.   What has Tesco been doing to deal with the situation? This is what the article says:

Last year, Tesco announced it would be spending £1bn on improving its stores in the UK, investing in shop upgrades, product ranges, more staff, as well as its online offering.

There are a number of flaws on data driven decision making. For one data driven decision making assumes that the future will be a continuation of the past.  Which is rather like saying all the swans that we have come across are white, so we should plan for white swans.  And then, one day you find that the black swan shows up!  The recession and the shift in consumer behaviour that resulted from this recession was the black swan for Tesco.

Furthermore, I hazard a guess that in their adoration at the pulpit of data driven decision making the folks at Tesco forgot the dimensions that matter but were not fed into the data and the predictive models. What dimensions? Like the customer’s experience of shopping at Tesco stores: not enough staff, unhappy staff, stores looking more and more dated by the day, the quality of their products ……

It looks like the folks at Tesco did not heed the sage words of one of my idols:

Not everything that counts can be counted, and not everything that can be counted counts.

– Einstein

Want to get value out of your data and analytics investment? Then deal with this issue before you buy the software.

‘Rationals’, data and the wonders of analytics

I think I have said this before and I will say it again: my fascination is with us – human beings being human beings.  In particular I am fascinated by ‘Rationals’.  Whom I am speaking about when I speak ‘Rationals’?  I am pointing at/towards my fellow human beings who pride themselves in being ‘rational’, ‘objective’, ‘scientific’ – they usually love order, logic, reason and are attracted to / come from ‘engineering-science-accountancy-economics-mathemetics” type disciplines.

What is it that I find fascinating about my fellow ‘Rationals’?  Before I go further I should point out that I used to be a ‘Rational’ and do get sucked back into being a ‘Rational’ if I am not being mindful.  What do you expect?  I studied Mathematics, Physics, Chemistry.  I have a BSc in Applied Physics and then went on to study accountancy and qualified as a Chartered Accountant.  OK back to my question: what do I find fascinating about ‘Rationals’?  Bluntly speaking, ‘Rationals’ are the most irrational people and they totally do not get this paradox, this joke.   Put differently, ‘Rationals’ are blind they are to the way that human beings work, organisations work, society works.  They are suckers for universal principles and insist that the world act in accordance with these universal principles.  One of these fundamental “shoulds” is that “people should behave rationally”.  Why does this matter?

Right now there is a fad in progress.  The people behind this fad hail, loudly and frequently, the numerous wonders (and benefits) of what data and analytics can do.  I suspect that many of them are ‘Rationals’ with a good sprinkling of ‘Marketing’ types thrown in.  They proclaim that big data and state of the art analytics (social, content, text, predictive…..) will light up our world and lead to the promised land: mountains of revenue; costs trimmed to the bone – everything working so efficiently so as to render void the 2nd law of thermodynamics (order to disorder) and the messiness of the real world; and an ocean of profits.  And all you have to do is to mine the Big Data!

The ideal world: how ‘Rationals’ assume the world works

The ‘Rationals’ assume that every person and certain every influential person making decisions is John Maynard Keynes (the famous economist).  What do I mean?  He was once asked how he responds to new data that did not support his earlier decisions and judgements.  JMK replied “I change my opinion. What do you do sir?”

Yes, that is the ideal.  Every little one of us as a perfect computer: taking in data as input, crunching that data against any number of dependable algorithms, spitting out the answer and doing what is in line with that answer ignoring our ‘points of view’, our ‘prejudices and bias’.  Now lets take a look at reality.

The real world: human beings are strange, marvellous, creatures

Daniel Kahneman in his latest book (Thinking, fast and slow) spells out that human beings are essentially a meaning making organism that thinks/works in stories and jumps to instant conclusions as long as the story fits the preconceived schema.  He writes “The implication is clear: as the psychologist Jonathan Haidt said in another context, “The emotional tails wags the rational dog.” The affect heuristic simplifies our lives by creating a world that is much tidier than reality….” Mr Kahneman has titled one his chapters “Causes trump statistics“.

Lets just ignore the fact that the people interpreting data and presenting it to decision makers are human and they exhibit the same ‘marvels and failings’ as the ordinary person: in test after test conducted by Daniel Kahneman and Amos Tversky the professional statisticians made similar ‘errors’ to the ordinary person.  By ignoring this you would think I have just spelled out an excellent reason to deploy analytics to drive decision making – to take out these human failings.  If only the world was that simple my ‘Rational’ friend.  Let me share three stories with you to illuminate the nature of the real world.

Story 1.  On page 116, Mr Kahneman tells the story of how Tom Gilovich and Robert Vallone did a statistical analysis of thousands of sequences of shots in basketball and concluded that there is no such thing as a hot hand in basketball.  Thats right according to the data and the statistical interpretation of data, there is no such thing as a basketball player having a hot hand – it is a human invention.  Now here is the instructive part, how did the basketball public (coaches, media, fans…) react to this conclusion?  Disbelief – just in case you don’t get that it means they did not believe it!  This is what Red Auerback, the celebrated coach of the Boston Cetics said: “Who is this guy? So he makes a study. I couldn’t care less.”

Story 2.  There is an amusing and enlightening story about human beings and it goes like this.  Once upon a time an ordinary villager died.  His body was cleansed, taken to the local mosque, the villagers gathered together at the mosque, the prayers were said and all the necessary rituals performed with the family present.  Then the body was put into the coffin.  Some of the villagers lifted the coffin on their shoulders and headed for the cemetry.  On the way they heard a knocking coming from the inside of the coffin. Then they heard a voice say “Let me out of here. Let me out of here. I’m alive, why have you put me in this box?  Let me out of here!”  The folks carrying the coffin replied “No, this is a trick.  You are dead.  We have said prayers and carried out all the necessary rituals.  Now it is time to bury you.”  No matter how much the villager pleaded to be let out the folks carrying his coffin refused to listen.  They were adamant that he could not be alive after all hadn’t all the villager seen his dead body, said prayers….. they would not be made fools of by the ‘dead man’ and end up being laughed at by the villagers when they returned to the village, opened the coffin and found that the the voice coming from the coffin was a hoax!  No, he was dead, everyone knew that!

Story 3. During World War II there was a notable disaster – Operation Market Garden.  This is what Wikipedia says, “Montgomery was able to persuade Eisenhower to adopt his strategy of a single thrust to the Ruhr with Operation Market Garden in September 1944….. the operation failed with the destruction of the British 1st Airborne Division at the Battle of Arnhem and the loss of any hopes of invading Germany by the end of 1944.”  What makes this relevant is the fact that this disaster could have been avoided.  How?  Brian Urquhart.  This is what Wikipedia says, read it carefully:

“In the autumn, as the 1st Airborne Corps Intelligence Officer, he assisted with the planning for Operation Market Garden, an ambitious airborne operation designed to seize the Dutch bridges over the rivers barring the Allied advance into northern Germany. He became convinced that the plan was critically flawed, and attempted to persuade his superiors to modify or abort their plans in light of crucial information obtained from aerial reconnaissance and the Dutch resistance. The episode was described by Cornelius Ryan in his book on “Market Garden”, A Bridge Too Far. (In the film version, directed by Richard Attenborough, Urquhart’s character was renamed “Major Fuller”, to avoid confusion with a similarly named British General.) ……… but he became deeply depressed by his failure to persuade his superiors to halt the operation and requested a transfer out of the airborne forces.

General Browning ignored the intelligence supplied by Brian Urquhart, the intelligence officer.  Why?  He didn’t want to have to tell his boss Field Marshall Montgomery that he was cancelling another operation – many airborne operations had previously been cancelled.  How did the operation turn out?

So what do we have here?  We have an intelligence officer doing his job and providing the intelligence.  The intelligence goes against what the top brass is committed to and the intelligence is ignored, the operation goes ahead and it is a disaster.  Do you think this is a one-off event?  It happens all the time: think Iraq and WMDs; think of the financial crisis and the reckless lending that led to it and the warnings that were ignored….  Does the Sufi story (told earlier) sound far fetched now?

If you want to get value out of data and analytics then deal with human nature as it is

I love zen – it says see life as it is and as it is not, leave behind your mind full of theories, concepts, projections of how you want things to be.   You would do well to act on that advice before you spend a fortune on “big data and analytics”.   Why?  It is not easy to get the right data (information overload as much of an issue as data quality and data integration) and convert that into useful intelligence that can drive decision-making.  No it is not easy.  Yes, I do know that the smooth tongued marketers are promising you that it is so easy.  The reality is that it takes time (lots of it), effort (lots of it) and money (lots of it).  Yes, the technical aspect is doable if you put in the time, effort and money that is required: the experts will come in and do it for you.

The hard part, the really hard part, is the human part.  It is dealing with, working with, human nature as it is. Human beings do not get data and do not value data – they simply do not relate to it like they relate to, say, a dog.  Do you really think that the people in your organisation appreciate the value of data and put in the time and effort to enter the right data into the right fields?  If you think that then you have spent far too much of your time in the Ivory Tower of the executive office.   Yet even that issue – of getting your people to enter the right data into the right fields/systems – is insignificant to the real issue.  What is the real issue?  Getting managers to give up their pet theories, their ideological convictions, their vested interests, their intuition, their past experience and use data and analytics to make decisions.  That is the central issue that you have to and should deal with.

My advice

Before you go and spend a fortune on ‘big data and analytics’ do the following:

Find out the total cost of ownership.  The set-up cost (people, technology, other) and the on going operational cost.  Statisticians don’t come cheap and then their are annual software licensing fees to think of…

Hold a ‘Big Data and Analytics’ party and invite all the key people from you business to that party.  Spell out the wonders that ‘Big Data and Analytics’ will bring them and the company.  Then ask them to pay (out of their existing budgets as they are) to attend the party.  The price of admission to this party?  Just divide the total cost of ownership (say over three years) between each of the players in the organisation.  Then see who turns up.  That might give a true picture of how much passion there is for ‘Big Data and Analytics’ within your organisation.  Or you can try the “build it and they will come” approach – your party, your choice!

Three reasons why converting data into valuable business results is no picnic!

Nancy Porte knows what she is talking about

I have been reading a PR release on the latest Verint-Vovici voice of the customer survey.   The key finding is “organizations hailing from a wide range of industries and company sizes are catching on to the value of the Voice of the Customer”.   And the conclusion is “research shows that as long as companies aim to enhance the customer experience, improve customer loyalty and increase profitability, the VoC will always fill a mission-critical role”.   It is good to see that an ex-employer, The Peppers & Rogers Group, has been involved in this study.  Let’s move on to the part of the press release that did catch my attention:

“People are amazing at collecting data, but they’re often less skilled at creating insights out of it and spreading them throughout the whole organization,” adds Vovici’s Porte. “Data is great, but it rarely means anything unless you’ve figured out exactly what that data is saying and what you’re going to do about it.”

What happens when you give a budding Alexander the Great a fleet of stealth bombers?

Imagine that you turn up at the residence of a budding third world ‘Alexander the Great’ – a dictator with dreams of empire.  You offer him a fleet of stealth bombers with all the associated armaments.  You are excited, the dictator is excited – you see yourself as rich and the dictators sees the world at his knees.  What you have both forgotten is that it takes highly skilled pilots to fly these stealth bombers.  And you need to put in place a whole ‘infrastructure’ (hangars, airstrips, fuel, missiles, technicians) to enable highly skilled pilots to make use of these stealth bombers.  You might be wondering what this has to do with VoC and Customer Analytics.  Everything.

There are plenty of vendors offering analytics stealth bombers (advanced data mining and analytics platforms, packages and ‘solutions’).  Before you embrace your dreams of ‘world domination through analytics’ you might want to consider and prepare for the following:

  • The fuel (data) that powers analytics is typically missing, insufficient and/or ‘dirty’;
  • The pilots (statisticians) that need to fly the stealth bombers (the analytics packages) are in short supply – there simply are not enough of them to go around to fulfill the dream that is being aggressively communicated;
  • The infrastructure (leadership, mindset, culture, practices, processes….) that is required to ‘make use of the skills of the pilots’ and ‘exploit the potential of the stealth bombers’ is simply not present in the vast majority of organisation and putting it in place is a BIG ask, which requires patience (think long term) and commitment (to face and overcome the hurdles – big and small).

What is so in the real world?

What is the state of affairs in the real world?  I cannot answer that question as I have not sampled the whole world of business.  Yet, I can share my lived experience with you when it comes to the analytics front.  Allow me to illustrate with two examples.

As a customer based strategist I need and generate insights in order to construct viable strategies.  In one consulting engagement I asked a question: how many customers did you sign-up last year over channel X?  Sometime later I was handed a report.  I got a different answer depending on which page I looked at.  There was only a small difference between the numbers and the point is that there should have been no difference!  Either the definition of ‘customer’ was not consistent over the enterprise.  Or the definition of the ‘channel X’ was not consistent.  Or the enterprise systems and/or people were not able to add up properly.  Sound bad enough?  What if I told you that the managers in this enterprise prided themselves on the quality of their management information?

In another consulting assignment (involving the formulation of a customer strategy) I was in the process of getting to grips with the customer base – specifically, customer profiling and segmentation.  I needed to understand who was buying a particular product.  So I asked a helpful member of the client organisation to come back with the answer.  I was handed a small document and it clearly stated that the buyers were male, in their thirties and living in the larger cities.  Having been burned more than once, I asked to see the actual figures.  Upon reviewing the figures I found that the buyers (customers) were indeed in their thirties and living in larger cities.  However, there was no statistical difference in gender: the buyers was just as likely to be female as male.  You might be wondering how someone who is trusted (and whose job it is) to interpret data get it wrong so badly?  I can tell you that this is no exception – misinterpretation of data is the norm rather than the exception.  Why?  Because we are not statistically minded.  Thinking statistically is arduous and is a skill that has to be learned and continually practiced.