Peter Cochrane's Blog: Damn metrics

Written at a public house just outside Morpeth, Northumberland and dispatched via a public wi-fi service from a service station on the A1 just outside Peterborough

About 20 years ago I was drawn into a public debate and enquiry on the state of education and educational standards. As part of the process I was interviewed by a board responsible for gathering industry views on the topic. This turned out to be a reasonably relaxed affair with three people on one side of a table and me on the other.

From the start of the discussion I was intrigued by one of the panel who had a huge spreadsheet (paper only in those days) in front of him. The conversation was about five minutes in and going well when the spreadsheet man jumped in with the words: "What you just said can't be right because I have the figures in front of me!"

After a couple more interruptions of the same manner I figured I had the measure of him - this was 'clipboard man'. To be specific: someone who uses numbers as a means of support rather than illumination.

So strong was this individual's belief in, and dependence on, raw numbers that he was unwilling to consider any alternative view. This I have always considered, rightly or wrongly, to be the start of management by metrics and death by numbers. More reasonably, I suppose, it was the point at which I became all too aware of the very real dangers of a metrics-dominated management mind.

Of course, in any organisation some form of measure is needed to gauge performance against stated objectives. Typically these might include income, expenditure, profit, resource utilisation and targets achieved relative to some agreed plan.

The number of metrics might reasonably number around five - in the extreme perhaps 10 - but certainly no more. And the reason for such a small number? Above that number metrics become meaningless due to their rapidly reducing significance and non-mutual exclusivity.

So what did we have in industry 20 years ago? Managers and teams were typically judged against metrics spanning less than five key parameters. And what do we have today? About the same!

And how about education and government 20 years ago? Managers and teams were typically being judged against metrics spanning greater than 10 key parameters. And what do we have today? Some government departments now 'enjoy' well over 100 metrics by which their performance is judged.

How do people manage to more than 100 metrics? They can't! So they fiddle the results at every level of the management chain. This practice is now widespread and endemic.

In education it's manifest in the sitting of examinations where the students are coached for specific question-sets, and the marking scheme is administered in a liberal way. In healthcare it is the classification of patients at the point they are officially diagnosed, recognised to be on a waiting list and treated.

Management by very large numbers of metrics always leads to a layered dishonesty where people do their job as best they can, and then report up what the system is demanding. If year-on-year improvements in examination results have been demanded, then by god such a system will be delivered.

Could it be any worse? Oh yes! How about spending money on IT that is designed to manage the metrics and make sure they all meet stated targets by year end? Such systems have been built and are a living testament to the view 'garbage in - garbage out'.

What a bastardisation of the capabilities of IT - and what an opportunity missed to address the fundamental problem in the management of a society.

IT has already changed - and can continue to change - everything. But it is up to us to make sure the change is a positive one for society as a whole. What IT cannot do is to take flawed concepts and practices and make them effective. However, it can perpetuate and amplify bad practices if it is used to support them and hide the true facts. The choice is ours!