The Naked CIO: The great recovery disaster

Many organisations have elaborate disaster-recovery plans. But how many plans are based on fragile theory rather than practical experience, asks the Naked CIO - and how many are stored on the network?

I recently had to review our procedures for what we would do in a disaster. On paper all the various bits and pieces were there. But I was concerned at the absence of practical experience behind the plans.

Exclusive column: The Naked CIO

See what this CIO really thinks…

The Naked CIO: Best backgrounds for CIOs?

The Naked CIO: Why boards get IT spend so wrong

The Naked CIO: Tech's weasel words

The Naked CIO: Poisoned BlackBerrys

The Naked CIO: Enemies of the state

Failover, data centre resilience and outside influence are all considerable factors in successful disaster recovery.

But testing production environments with today's 24/7 uptime is near impossible. Just mapping the organisation's IT real estate is in itself a complex and confusing exercise.

Even in isolation the use of third-party services and applications is mind-boggling. In an internal control environment they communicate and extrapolate data internally and externally in fractions of seconds.

Couple those complications with alternative protocols and actions, such as power and communications failure, satellite office accessibility, complete or partial loss of one or more critical services such as telecoms or internet and it is impossible to predict or test the practical response to these very real potential problems.

So how do we confidently stand behind procedures that in most businesses remain based on theory? How do we assure our business disasters have been planned for if we have no real way of determining if the contingency measures will work?

I remember the madness after the London bombings in July 2005: no mobile service to contact colleagues and family and to co-ordinate efforts. Even though there was no disruption to my systems during that outage it is the only experience I have had to assess potential difficulties in a time of emergency and chaos.

Communication is the first thing to be hit in a crisis and yet the last thing to be planned for in a disaster-recovery plan.

Many plans I've read talk about back-up, storage, data centre failover, redundant systems but rarely mention who does what and when or how communication should be managed in the face of disruptions.

What we have to decide is whether a disaster-recovery plan is just a piece of paper management can hold up and say "We have one!" or whether it is a realistic, practical plan that can assist a business to maintain functions in a crisis.

How many organisations have their disaster recovery plan stored only on their network and would have trouble accessing it if systems were hit?

My approach has always been to have a simple action plan and protocol that can be followed no matter what the disaster. Do not make the process more complex than necessary by having multiple responses to multiple threats.

Additionally, focus on communication, roles and responsibilities - who does what in that event. Experience, skills and brains are the best fallback for achieving a quick response because there are no textbook definitions of disaster. Nimble reactions to unpredictable circumstances are the most effective answer.

I can offer no solution to the puzzle of how most businesses should test the practical nature of their theoretical plans. But it is important to have some confidence in at least the initial steps to be taken in a disaster.

Crisis response teams, command and control, communication protocols, data protection, back-up and restoration and critical communications systems should all be elements that, if not tested, are at least frequently discussed and communicated.

We never want a disaster, that much is certain. But if we have one I certainly never want to be in the headlines for the wrong reasons.