Devil's Advocate: To trust or not to trust machines?

Martin Brampton examines why humans sometimes feel comfortable relying on machines and computers - and why they sometimes prefer to see the decisions made by people.

We have a very strange attitude to machines. They are essential to modern life and we rely on them to a remarkable degree. Yet we are also reluctant to believe them in many instances, or to accept their 'judgements'.

Only recently, the nuclear reprocessing works at Sellafield suffered a major leak. The material involved is highly radioactive and dissolved in nitric acid. The leak resulted in a rather large puddle of around 83 cubic metres of this nasty stuff. Much of it is supposed to be returned to its originating country after processing, a task that is rendered impossible when it is mixed up in unknown proportions.

The curious thing is that the plant's instrumentation showed at least 100 warnings over six months, indicating that it had sprung a major leak. The instruments were not believed because it was thought that such a new plant could not fail so soon. In fact, this assumption was mistaken; errors in construction had drastically shortened the life expectancy of some of the pipe work.

Indeed, accidents quite often happen because human beings do not behave like machines. Airline pilots have got into difficulties through not believing their instruments. Once a person has formed a view about some situation, the evidence tends to be interpreted so as to support the initial assumption. Even when they are contradictory indications, they tend to be dismissed as faults.

In fact, the automatic systems have to be pretty good. Aircraft can take off and land in weather where the crew cannot see where they are going. If the instruments cannot be relied on, then the passengers are at risk. And yet, we like to take comfort from knowing that the crew can override the automatic systems, even though the system may be right.

The lack of automatic systems has also been a cause of accidents. The view that people can be expected to always get things right is unrealistic. It is natural and inevitable for people to make mistakes. So trains will from time to time go past red signals. Unfortunately, the results can be disastrous. But anybody involved with safety in that kind of environment is aware that the only solution is to accommodate human error, not to attempt its elimination.

At a less life-threatening level, many decisions once made by bank managers are settled by computers. We fear that this dehumanises our transactions, which of course it does. But the evidence suggests that it also makes for more consistent decision making. Real live managers are notorious for making poor decisions at certain times, notably on Friday afternoons. Computers normally come up with the same decision on the same data every time.

So does this mean we should always put our trust in machines? Sadly there is no simple conclusion. In many situations, we are probably better off relying on the machines, especially those that are dedicated to a specific task. Their capabilities have often been carefully refined through extensive testing and validation.

But there is always the risk that the human thought behind the design of the machine has been based on wrong assumptions. The failings that afflict nuclear plant operators, airline crew and train drivers also afflict system designers. We all hold on to assumptions that would not stand up to examination if only we could examine them in a fresh light. As with so many things, we are stuck with the conclusion that there is just no easy answer.