Some CIOs are imagining potential disasters that go well beyond the everyday hiccups that can disrupt applications and networks. Others, recognizing how integral IT is to business today, are focusing on the need to recover instantaneously from any unforeseen event. Many are trying to do both. But CIOs agree that disaster recovery planning has taken on an immediacy that didn’t exist in the ’90s.

And they expect the threats to get worse. “The things you think about are, What will the virus/hacker people be able to do 10 years from now? What do I need to do to keep my capabilities ahead of the game?” says Rob Reeg, senior vice president of global operations at MasterCard International Inc.

Old Worries and New

When it comes to disaster recovery, the concerns are diverse. CIOs say they still worry about the traditional problems, from those manual errors and little snafus that can crash a system to natural disasters like fire and flood. But they’ve also added new concerns that range from catastrophic power loss and network attacks to employee sabotage and terrorist attacks.

Raj Sampath, chief technology officer at LoanCity, a wholesale residential mortgage lender in San Jose, has considered just about all those scenarios. He says his biggest fear is a hacker attack. “It’s the unknown part — I don’t know how or when it’s going to be,” Sampath says.

He says a successful attack could not only corrupt his system but also compromise the personal data of the company’s customers. That’s why he has a firewall manager — a combination of hardware and software that acts as the first point of contact for the external world, manages the security of the company’s systems, protects servers from hackers and allows only certain specified transactions. Sampath also diligently keeps security software updated.
He worries about other scenarios, too, such as earthquakes knocking out his primary data center and employees downloading infectious programs. So he sets up redundant systems, continually updates antivirus software, monitors employee computer use and uses technology from San Jose-based Sonasoft Corp. that automates the backup and recovery process for Microsoft Exchange and SQL and Windows servers.

Other CIOs draw their new list of concerns from current events such as the 9/11 terrorist attacks and the August 2003 blackout that affected the Northeast. They ask, “What if someone sets off a dirty bomb? Or launches a bioterrorist attack? What happens if the country’s aging power grid fails?”

“It’s a different world. There are so many more things to consider than the traditional fire, flood and theft,” says Robert Rosen, a Bethesda, Md.-based CIO in the U.S. government and president of Share Inc., an IBM user group.
For example, as he toured a disaster recovery site last year, Rosen was impressed by its meticulous planning and features. Still, he was concerned when he heard a low-flying plane overhead, noting that a site’s proximity to an airport — even a small one —means there’s an increased risk of it being hit by a crashing aircraft.

But that doesn’t mean CIOs have to prepare for every scenario they can imagine. Companies usually make their disaster recovery decisions based on cost and risk analysis, says David Palermo, vice president of marketing at SunGard Availability Services LP in Wayne, Pa. They prioritize risks according to the likelihood of various scenarios and the effect each one might have. “At some point, you’re out of money and you have to make your choice,” Palermo says.

A key to risk analysis is that it’s not always about full-blown system failures; even small problems can have significant consequences.
“Disaster has taken on new meaning in this era of Sarbanes-Oxley and all this government regulation,” says Mike Kahn, managing director of The Clipper Group Inc., a technology acquisition consultancy in Wellesley, Mass. If regulators come knocking, they now expect companies to produce all the required data within hours — not weeks, as they once did, he says.

Speedy Recovery

The best-prepared executives recognize that speed is essential in recovering from whatever disaster might come to pass, Kahn says. That’s why there’s a trend toward enabling technology users to restore their own documents, so a lawyer, for example, can retrieve a brief that took weeks to write but an instant to accidentally delete.

And in addition to regularly backing up to tape that’s then stored off-site, companies are employing newer technologies to take snapshots — every five minutes, or every hour, depending on the business — to reduce the risk of potential loss, Kahn says.

But while the technology exists to ensure that a company doesn’t experience a catastrophic loss of data even if its systems go down, experts say executives need to make disaster recovery a priority — and fund it appropriately — if they want to guarantee business continuity during almost any situation.

Robert Borr, CIO at Quincy Medical Center Inc. in Quincy, Mass., has tried to protect his organization from just about any disaster that could take out his systems. In addition to strict backup procedures, he has agreements with hardware vendors to deliver new equipment within 24 hours if needed.

He pays extra for that but says, “If we can get computers delivered within 24 hours, we can be up and running seven or eight hours after that.”
Others have taken even more aggressive approaches to guarantee that they’re prepared for anything. Agnoli says his law firm increasingly focuses on building in redundancies to ensure that if an office system is taken out or an e-mail server goes down, another one can quickly take over, averting a crisis while the problem is being fixed.

“The goal,” Agnoli says, “is to avoid ever having to get into a disaster recovery situation if we can.”