Sunday, 24 February 2008

How well do you know your company? (my guess is not very well at all…)

In the 1970s, there was a series of academic studies which looked at managers’ perceptions of the “volatility” of their company’s business environments. These studies all found that different managers within the same organization usually had widely varying views on how volatile their business was: the correlation between different people’s assessments was virtually zero.

In addition, these studies found that there was hardly any relation between objective measures of business volatility and managers’ estimates of these measures (if anything, the correlations were negative; managers in stable environments thought their business was relatively turbulent and vice versa). Similar results were obtained for other variables. The researchers concluded that managers’ perceptions of their own businesses were usually plain wrong.

Two business school professors – John Mezias and Bill Starbuck; at the time at New York University – set out to examine this claim further (and published their results a few years ago) simply because they found it hard to believe; at least at first…

They thought “‘business volatility’, that’s a bit vague and abstract, let’s start with something simple” and they asked executives from a wide range of companies to tell them what their business unit’s sales were in the previous year. Then they looked up the units’ actual sales figures. The average answer was 475.7% wrong. That’s 475.5%...! Sales of their own business unit!!

Then John and Bill thought, “perhaps we should pick something they find really important”. So they approached a blue-chip company and asked them what the company’s absolute top-notch priority was: the CEO declared that the absolute top priority throughout the entire company was “quality improvement”.

And indeed, they had put their money where their mouth was: many managers attended quality improvement training courses, each division had a dedicated department focusing on quality performance and the company had developed various quality metrics. Furthermore, all managers received quarterly quality improvement reports, and 74% of them indicated in a survey that they expected to receive large increases in their personal rewards if their divisions managed to increase quality. Yep, “quality” was important to them!

Quality was measured in the company, following the specialist training techniques, in terms of “sigma” (a measurement of the error rate). When John and Bill asked the managers what the sigma of their department was, the average error in their answer was…(wait for it)... 715.1%. A whopping 715%! They really had no clue…

Note that these had even been managers brave enough to give any answer at all; 7 out of 10 managers, when asked, had refused to give any estimate, declaring “I don’t know”. It seems likely that they had realised they had no clue and rather than make a complete fool of themselves, they opted not to say anything.

Granted, when John and Bill finally asked the brave ones who dared to give an answer to express their unit’s error rate not in terms of the illustrious “sigma” but in the plain human terms of “what percentage of products have errors?” they did a lot better: almost 7 out of 10 managers managed to give an answer which was less than 50% off the mark.

Of course these people are not all fools. They are usually smart, well-trained and hard-working. It is just that they have no clue about the numbers describing their own business – and managers usually don’t. We spend a lot of time, money, effort and attention quantifying all sorts of aspects of our organisations but, at the end of the day, make decisions ignoring all these numbers, using our experience, qualitative assessment and gut instinct.

And that’s probably for the better; if we’d base decisions on our (alleged) knowledge of the numbers, we’d be prone to not only shoot ourselves in the foot, but also in the chin, the head, the back-side and the bodily parts of several of our neighbours.


Prem Rao said...

Numbers can also be expressed in terms of our convenience,can't they? I can speak of a 150 % increase without mentioning that the base figure was ever so low!

Mark Twain popularized a quote attributed to Benjamin Disraeli:"There are three kinds of lies: lies, damned lies and statistics"!

Prem Rao

vlade said...

Apologies for ranting here in comments to your posts, but it seems you're picking subjects close to my heart.... Namely, what you allude to at the end of your post - basing (or not) our decisions on models.

As someone in finance industry, I think we're placing way, way too much trust in models without realizing that model is not reality, and it has assumptions that are all too easily broken. I think the mark of a great manager is to realise this, and have a "gut feel" to go with your model(s) (so, amongst other things you know when the model's failing). We're trying to simplify too complex processes and too often forget that it's just a simplification, not reality.

Freek said...

Don't get me wrong, I love numbers! Much of my own academic research is highly quantitative, using elaborate statistical techniques. Yet, indeed, I strongly believe that part of working with numbers is making sure that they don't lure you into a false sense of security, and that you see their limitations and potential dangers, for instance of making one myopic. I will undoubtedly not be able to resist writing more on this topic in the (near) future.

Xian said...

This topic triggered my memory of the chapters in information system, namely business intelligence and "user stories" and at the end of it, I concluded that we really sometimes, if not often, trying a bit too hard with numbers. As someone currently involved in retail merchandising, seeing it in the workplace feels almost voodoo, it makes people say "odd" things despite their experience. I share the same feeling as the last paragraph. After all, people are paying big money for people with experience, why shouldn't they use what they are paying for? Or may be I am the odd ball?

Xian (a maths grad that is still in love with number!)

vlade said...

I actually work in a quantitative area, so numbers are my bread and butter - but also becasue of that I can see the large immediate impact believing your models are the reality (as opposed "can help to understand reality") can have (including a huge financial impact).

Paul said...

I think managers focus on what they can do something about - they recognise that they contribute to the overall departmental figures but there are just so many other factors beyond their control - better to focus on their own personal targets. Why should I know or worry about the big numbers.

I'm not saying that is correct - but the ignorance may not be rooted in a lack of quantitative understanding.

business opportunities said...

I agree with you Paul. It will lead to not knowing.

Managers oversee that's why they should know every details of everything.