Disaster recovery plans must address a wider range of risks than ever before, reports Russ Banham.
Amateur golfer and professional disaster recovery specialist Greg Fennel is a meticulous planner, whether it is lining up a chip shot from 10 feet off the green or running business continuity systems at Levi Strauss & Co, the well-known maker of blue jeans. "I like nothing more than logically planning something out, step by step, from beginning to end," Mr Fennel explains.
Such care and caution is just what the San Francisco-based company needed nearly a decade ago, when a small earthquake shut down its communications operations. "We were able to get our data, but we could not get into our buildings," recalls Mr Fennel, a 17-year Levi Strauss veteran. "What good is data if you cannot access it?"
A good question. Down and out for three days, the company's patience was stretched as tight as a pair of its just-washed 501 denims. By the time it came back on line, the company estimated it had lost nearly a quarter million dollars in sales, not to mention the indirect costs of a dented reputation. Today, the price tag would be far higher. "If the same disaster were to happen today - and we were not prepared - it would cost us about a million dollars a day - easily," Mr Fennel says.
Fortunately, Levi Strauss is ready should lightning strike twice. Like many large manufacturers and financial services companies, the company has invested in comprehensive disaster recovery plans and tools. If another earthquake (or a fire, union strike or massive flood) struck the company, and its primary communications and/or distribution outlets shut down, all its data is backed up by an off site computer system. Moreover, its work force is rehearsed and ready to move to an alternate location if need be.
"We have our customer credibility and reputation at stake here," Mr Fennel says. "If we tell our customers we are not going to be able to ship our product for 10 days or so, we lose our brand image credibility. And that drives right down to the bottom line."
Some companies lose more than just their brand equity when disaster hits. A study by the University of Texas a couple years ago indicated that 43% of the companies that do not back up their computer files at an alternate site and experience a data loss never reopen for business.
Other studies paint a similar portrait of gloom. According to the Journal of Systems Management, a typical manufacturing and distribution company with more than $200 million in annual sales stands to lose more than $100,000 after four days without information systems services. If they are still not up after 10 days, the loss is more than $1 million.
Despite these caveats, some companies bet the odds. That may be a mistake. "The old phrase, 'It can't happen here,' no longer fits in today's world," says Pat Moore, vice president of business continuity education at Strohl Systems Group, Inc., a King of Prussia, Penn-based company in the disaster planning software and services field.
"Nowadays you must plan for eventuality, not just probability."
While the business of business continuity has been around since the late 1970s, its more recent incarnation is far broader, contextually, than the computer data recovery companies that existed then. Today's planning elite are more apt to address a wider vocabulary of risks in their processes. "There has been a tremendous expansion in emergency response plans to address organizational-wide continuity of operations and planning," Ms Moore says.
"Historically, contingency planning was based on disaster recovery and resumption, which typically addressed bringing up critical applications in ever-shorter time frames. The focus today is much more on all operations and processes that affect revenue-generating business units."
Disaster recovery plans used to be written to address natural and man-made disasters, Ms Moore notes. "Now they are being written to address terrorism, loss of competitive advantage or market share, union grievances that cause the shutdown of operations, loss of key executives and both pre-loss and post-loss trauma counseling. All of these directly affect the bottom line immediately or will at some later date," she says.
Strohl's new business impact analysis software, called BIA Professional, reflects the trend toward more systemic business continuity planning. The new software doubles the number of survey questions that respondents answer to determine their risks and recovery objectives.
Comdisco Inc, a Rosemont, Ill-based competitor of Strohl's, also recently unveiled a new business impact analysis product. "It is a great tool for management to examine the impact of a disaster - money-wise - in each business segment," says Mark Avery, a vice president in Comdisco's software planning group.
He agrees there are more than just earthquakes and floods threatening the continuation of a company's business operations. "Not only must recovery plans address a wider range of risks, they also must be truly global," Mr Avery says.
Once a company gets its ducks lined up as far as the potential impact of a disaster, there are generally two business contingency strategies it must take: establishing an off-premises "hot site" for employees to continue normal business operations and incorporating a back-up data information technology system for employees to use once they resume operations.
"Disaster recovery is more than data recovery," Mr Avery explains. "It is criti-cal that people in each business unit have both essential information systems and equipment to continue the flow of work."
Levi Strauss has two separate plans in place to address both data recovery and operations recovery - one purchased from Strohl and the other from Comdisco. "To recover our mission critical and time-sensitive business processes at our alternate space, we have a contract with Comdisco," Mr Fennel says.
"Each year we run two to three exercises in which our mission critical business units recover processes at the alternate space in Texas. Our data groups also participate in these exercises, recovering the client servers and networks."
Comdisco offers such recovery facilities in more than a dozen locations worldwide - even such remote places as Kuala Lumpur. Alternate sites range from 40,000 square feet mini-buildings to warehouses exceeding 160,000 square feet able to house as many as 100 corporations at one time. All of them house multiple power and communications feeds.
While Comdisco provides the company's data recovery, Strohl Systems provides its business planning recovery. "Strohl's logistics teams looked at what I needed to do to set up the business units at the other location," Mr Fennel explains.
"This was more about planning and the actual set-up of the alternate location. People do not just walk in and call someone up on the phone. All the extension numbers, where people sit and the infrastructure for seemingly simple things like office mail, have to be solidified."
Overall, the services provided by both companies cost Levi Strauss less than $200,000, a "pittance," Mr Fennel says, "when you consider how much we can lose directly and indirectly without this level of preparedness."
There are other reasons besides common sense for a company to invest in disaster planning. Wall Street analysts, for example, increasingly examine a company's plan of preparation for a variety of disasters, and value the company accordingly.
An unprepared company also may be liable to shareholder lawsuits in the event a disaster affects its stock performance. "Many senior officers and directors of corporations do not want to be found negligent for not investigating and using appropriate disaster planning and recovery," Mr Fennel says.
"We feel pressure from our board all the time. That is why we have determined very carefully our business requirements in a disaster and broadcast these on a regular basis." The company stipulates that it will not tolerate more than a 36-hour shutdown, he notes.
Some companies, especially those in the telecommunications and financial industries, are required to maintain business continuation plans. US banks, for example, must comply with the 1983 Banking Circular 177, which mandates contingency plans for reducing the impact of losing data processing. Additionally, the US Federal Reserve requires companies transferring more than $20 billion per day in funds to provide evidence of their ability to recover from a disaster within 24 hours.
The return back home
Another important element in disaster recovery planning is the return back to the original work site. Assessing the integrity of software and communications systems after a disaster and preparing them for the return of the work force takes time and money.
Fortunately, there are companies that can oversee such demands while management concentrates on maintaining ongoing business operations post-disaster. Bellcore, a Morristown, N.J.-based company, will determine after a disaster whether or not the phone lines, for example, are worth the cost of repairing or, instead, are more suited for replacement.
After the devastating floods in the midwestern United States last year, Bellcore was hired by US West to examine the viability of its communications systems. "They wanted to know if their plants had been impacted by the flood water and could be simply cleaned, rather than replaced," says Barbara Reagor, Bellcore's senior director of risk management solutions.
"We assessed the damage, directed the best course and then provided various methodologies for them to follow in the future should a similar disaster occur."
Bellcore is frequently hired by insurance companies seeking verification that a piece of communications technology is beyond repair, Ms Reagor says. "We will come in to look at equipment that is, say, wet from a flood, and provide an in-depth technical assessment of whether or not it can be salvaged," she explains.
"The insurer can use this information in a legal case or as a basis to resell the equipment. In either case, the total value of its loss is reduced."
Caught up in the net
Expect this year for major business continuity companies to begin offering their planning and recovery software over the internet. Both Comdisco and Strohl say such systems will be in place within the year. "Nobody has done this yet, but that is certainly where we are headed," says Mr Avery.
Once Comdisco unveils its web-based planning tool, set to debut in July, Mr Avery says customers will be able to develop multi-threaded recovery plans. "It will allow the building of recovery plans for different situations," he explains.
"A fire is different than an epidemic or a chemical spray condition, yet current systems generally do not distinguish between them. We have learned that different disasters typically require different recoveries."
The new web-based tool allows customers to build different disaster threads through easy-to-use wizards. "Each wizard indicates what a future wizard will be," Mr Avery explains. "If in the first level wizard you say the disaster is 'building damage,' that dictates the next dialogue box. You build from there."
When a disaster strikes, a company keys in the type of disaster, then clicks its way to the right recovery response, he adds.
The cost for the new web-based model is roughly $250,000. Says Mr Avery: "Some people freak out when we give them the price, but for Fortune 500 companies, which this system is geared for, that is certainly reasonable. Obviously, this isn't for the neighborhood car dealership."
Russ Banham is a journalist specialising in insurance and technology.