Earthquake fears make for extreme business continuity

On 18 April 1906 San Francisco was rocked by an earthquake which destroyed large swathes of the city and claimed the lives of more than 3,000 inhabitants of the Bay Area.

One resident, the author Jack London, wrote at the time: "San Francisco is gone. Nothing remains of it but memories... Its industrial section is wiped out. Its business section is wiped out. The factories and warehouses, the great stores and newspaper buildings... are all gone."

A century later, a bigger, bolder, rebuilt and more resilient San Francisco is more important than ever. Now it serves as the heart of the global IT industry and a major world financial centre. But it remains well aware of the terrible potential that exists along the San Andreas fault.

The vast skyscrapers downtown may now be built to withstand huge pressures but what about the infrastructure and the systems which keep modern business ticking - and the people who must be able to access them? Business continuity and disaster recovery are serious issues for all organisations in the Bay Area.

Barry Cardoza, head of business continuity contingency planning and disaster recovery at Union Bank of California, which is based in the heart of downtown San Francisco, told silicon.com earthquakes are a major consideration in the area but only as part of a complex patchwork of worst-case scenario planning - which includes everything from pandemics to terrorist attacks and acts of civil disorder.

Cardoza said: "You have disasters that you can see coming and you've got disasters that you can't see coming and an earthquake is an example of [the latter]. And you don't know how bad it's going to be until it hits."

As such, the bank must have processes in place ahead of such an event to mitigate the threat. Simply reacting is not a strategy. The contingency department must also understand every aspect of the business and weigh up downtime for each in terms of financial and reputational damage.

Stock traders, for example, might need to be at their most active in the wake of a disaster.

Cardoza said: "If they can't react when the markets are impacted then that could put you out of business. They might normally be turning over millions but the losses could be in the billions."

Multiple data centres in diverse locations, mirrored sites which can take over at the flick of a switch, "hot sites" - where staff can walk in and start working exactly as they would if they were in their normal location - and a vast amount of redundancy are also key to business continuity planning, said Cardoza, who is a member of the Bay Area Recovery Coalition with a number of peers from other major banks.

He added: "We have real-time mirroring between data centres. It's no longer a question of how long will it take us to back up from tape. It's a matter of minutes not hours once we switch over."

Communications lines are also critical, he said: "If we can't communicate with one another then we can't mitigate the problem."

Senior execs all have locations at their disposal which are fitted with video conferencing facilities among other communications options.

"There is a lot of redundancy," said Cardoza, showing three desk phones which all connect via different lines. "If we can't go one route then we can go another. We have redundant carriers and redundant phones."

He said the bank is also looking at satellite, BlackBerrys, walkie-talkies and SMS as communication alternatives as well as at sharing initiatives across the whole organisation - which is headed by Tokyo-Mitsubishi Bank, an institution that must weigh similar tectonic concerns for its own region.

The only benefit afforded in earthquake planning is that which is known: where the fault lies and what areas it affects. Fires, tornados and hurricanes tend to be bound by less clearly mapped lines.

"I'd rather be here than in tornado country, or in the way of a hurricane," said Cardoza.

Cardoza said as such the bank is able to plan recovery sites outside the areas which would be affected - though he conceded nowhere is safe from some seen or unforeseen problem.

"A business recovery site must be as far away as possible but not so far that people won't go there. And you want hardware to be as far away from the affected area as possible. It might not be a case of having site A and site B, you may also have site C and so on."

However, Cardoza said while he hopes his business continuity planning will never be tested, he said it is far from a bottomless pit for funds. He said much of the planning offers day to day strategic value, such as using data centres for load-balancing.

He said one important development, bringing in Citrix, whose product ensures all staff can work from home if locations became inaccessible for any reason, was actually a result of execs requesting to work from home in order to improve efficiency.

Cardoza told silicon.com: "It was only once I was speaking to Citrix I realised this is one of the greatest opportunities for resilience.

"We're looking for a situation which is win-win from a strategic point of view. I certainly never want to see these plans put to the test."

Another Bay Area business which must plan for any worst case scenario is FCE Benefit Administrators, which handles healthcare claims. Tom Toot, IT manager at FCE, told silicon.com he must balance the reality of the fact an earthquake would present issues of business continuity and disaster recovering at a time when his company's services might be most in-demand.

Toot said: "A few missed emails could prevent treatment being approved in a timely manner. I really don't want to say it's a matter of life or death but it really could be."

"We need business continuity. It is critical for us," he said.