Disaster Recovery: A Blueprint For Better Response

By Peter J. Brown

Recovery. Restoration. Continuity. These are not new buzzwords. They have taken on new meaning, however, as a result of the intensified planning and preparation underway throughout the past year. This effort is intended to yield quicker and more effective responses across the board to large-scale disasters, natural and otherwise.

The satellite industry is standing by with an instantaneously activated and sustainable communications solution. At first glance, providing back-up links and Business Continuity Services (BCS) may not appear to be highly specialized offerings; but, in the stressful environment where these services must flawlessly perform, the demands placed on both platforms and networks are enormous.

Thus far, it appears the private sector is proceeding at a much faster clip in terms of placing the right tools at the disposal of customers so that they can stay on track, but what exactly will result when dozens of back-up networks kick in simultaneously remains a mystery. Will contracts and protocols hold firm so that any unresolved issues surrounding coordination, prioritization and timely access to adequate capacity are not transformed into instant bottlenecks at a national level when the next big emergency happens? That is one of the big questions.

"I totally agree. I believe the space segment and ‘hub’ earth station resources must be either fully dedicated or available on demand to properly support first responder and disaster relief communications. These resources cannot be shared with public or commercial users unless an automatic prioritization capability is provided," says Consultant John Whetstone, president of JW Communications in Ellsworth, ME.

Whetstone recently designed a system for The American Red Cross involving a combination of flyaways and vehicular systems, including AVL MVSAT 1.2-meter antennas mounted on nine Ford Excursions. All of the remotes feature a boom-mounted Advantech 4-watt SSPA for up to 512 kbs back to a 3.8-meter antenna at Red Cross headquarters.

"At the very large level–big OC-3 and above type circuits–terrestrial infrastructure is required, and with these terrestrial-only solutions, last mile and single point of failure, Points of Presence will most certainly be a factor in a big emergency," says Jeff Gross, general manager of CA-based Immeon Network LLC.

"This, of course, is why satellite technology is so important, at least for rates of T3 and below with today’s Ku-band systems. Satellite coverage is ubiquitous, the bandwidth is instantly available and it can be directed/redirected without any last mile issues," Gross adds. "By offering network restoration for circuits at T1/E1 type rates and below, we eliminate the bottleneck issue and provide a critical service to our BCS customers."

Beyond The SNG Model

The similarities between disaster recovery services, BCS and satellite newsgathering (SNG) are well known. Each taps occasional use capacity with an emphasis on rapid activation more often than not in less than ideal circumstances.

"BCS is more about true network connectivity, while with SNG, you have a capability that can be deployed very quickly, has great standards-based IP networking capabilities, and utilizes proven technology," says Gross. "SNG truck operators have very big video pipes, albeit asymmetric, and they are increasingly utilizing non-real time digital communications, along with enhanced video capabilities, to improve the product they generate. At the same time, they are lowering their costs."

While satellite bandwidth costs more than terrestrial bandwidth in most instances, Immeon is driving the cost down to make the monthly fee for BCS more affordable. Having a cost-effective solution with completely diverse connectivity is certainly a very attractive alternative to a total meltdown of a corporate or governmental network. Bandwidth on demand is a necessary ingredient here, when it comes to reviving large multipoint networks quickly and reducing overhead. "This is the key concept. You have the bandwidth you need when you need it, and you only pay for what you use," says Gross.

Bandwidth on demand is a necessary ingredient here, when it comes to reviving large multipoint networks quickly and reducing overhead. "This is the key concept. You have the bandwidth you need when you need it, and you only pay for what you use," says Gross.

Immeon offers diversity-routed connectivity in either a full mesh or a hub and spoke configuration. Viasat’s Starwire Demand Assigned Multiple Access (DAMA) VSAT solution is used for full mesh links between all sites, while Linkstar with return channel rates up to 1 Mbs is used for hub and spoke systems.

"With Linkstar, we give the customer the choice of operating a virtual private network over the Internet, or operating over dedicated terrestrial circuits into our hub where Immeon hosts their dedicated router," says Gross. "Most of our Linkstar BCS customers are taking advantage of this dedicated router offering along with a dedicated line to our hub.

"With Starwire’s full mesh solution, the customer can bring the traffic directly from their remote sites to their central site using their own mini-hub. This is cost effective and eliminates the need for any terrestrial component," adds Gross.

Service level agreements and a Committed Information Rate (CIR) are parts of the overall package in a symmetric configuration. "A low-rate return channel without CIR will not do the job here in most instances," says Gross.

In terms of disaster recovery simulations, adequate testing of the specific applications that are most critical to customers will reveal if the applications will operate as advertised during an actual disaster.

"It will also expose any router configuration issues," says Gross. "Since every customer has unique network requirements, there are things you have to watch out for. This includes the effects of software upgrades and corporate network configuration changes, which may occur from time to time. Testing on a regular basis makes sure these issues do not adversely impact you at a critical time."

Addressing Cooperation And Coordination

While much lately has been made about how we might address things such as interoperability, large-scale disaster recovery is still a work in progress. The satellite industry can only do so much in light of the enormous complexity of the situation. David Beering, principal of Chicago-based Infinite Global Infrastructures LLC, sees public safety officials nationwide as shackled by a lack of satcom expertise and overwhelmed by all the unsolicited proposals they have stacked on their desks. Beering is advocating a different approach based on multiple cooperating municipalities sharing critical communications infrastructure, among other things.

"They would realize cost savings by buying services jointly rather than individually, support each other during large-scale emergencies or disasters and cross-train on new technologies as they become available for deployment on the shared infrastructure," writes Beering in a white paper he co-authored in January with his brother Peter, who serves as the counterterrorism coordinator for the city of Indianapolis.

Together the Beerings have been instrumental in forging a dialog between Chicago and Indianapolis involving the Chicago Office of Emergency Management and Communications and the Indianapolis public safety community. The City of Houston is also exploring a cooperative engagement, according to David Beering, who is designing and implementing a large Ku- band satellite hub facility at the Institute of Biosciences and Technology on the Texas Medical Center Campus in Houston for emergency communications applications.

"The problem is a relatively uneducated community in need of a course in Satellite 101, which covers hubs, remotes and space segment mapped to a terrestrial infrastructure with mobile outreach capabilities," says Beering. "What is really going to work here? The public safety community is swamped by the range of capabilities and thus is finding it hard, if not impossible, to answer this question."

The lack of an over-arching architecture at the national level or a sense of broader framework is only compounding the problem, and impeding the effectiveness of the collective attempt to find a solution. "Everyone is saying they are standards based. The level of integration and interoperability is a concern as well," says Beering.

DAMA: Ideal For Disaster Recovery And BCS

The concept of DAMA, which is based on the reusing of communication resources, is well understood in satellite circles, but not often so well understood by the users themselves. "It would be unthinkable to call someone and then leave both phones off the hook 24-7. This, however, is how most satellite systems operate with dedicated equipment such as modems, etc., transmitting dedicated carriers, full-time," says Whetstone. "This only makes sense if the bandwidth is being used full-time, which seldom happens. Enter DAMA. Share the equipment and bandwidth, just like hanging up the phone and calling someone else.

"All of the shared systems including cellular POTS (plain old telephone service) and two-way radio are really DAMA systems. The Hughes Direcway, Tachyon and iDirect are also DAMA systems," Whetstone adds.

Properly designed systems with Quality of Service (QoS) taken into account rarely get overloaded, according to Whetstone, who notes the increased cost because more bandwidth is allocated and goes unused until peak times.

"The cheaper systems suffer from frequent overloading. TDMA systems don’t generally refuse service, they just slow down. Priority (QoS) can be used, but again, this increases cost," Whetstone says. "With FDMA, it is more likely that a user will be refused service altogether if a frequency slot isn’t available."

Whetstone describes TDMA with IP traffic as really just a statistical multiplexing or stat mux platform in disguise, because the packets are transmitted based on priorities established within the router so that user datagram protocol traffic goes first, and then transmission control protocol next, etc.

"This allows VoIP and video teleconferencing to transmit first, which is appropriate since they are both real-time traffic. All IP packets flow on the same outbound carrier and are received by all remotes," says Whetstone.

"There is at least one FDMA DAMA system, NSI’s FlexiDAMA, that dynamically adjusts the SCPC carrier’s data rate and thus this system would behave more like TDMA, because users would be assigned frequency slots but would not be able to increase data rate causing a slowdown of their data transfer," Whetstone adds.

So, are DAMA systems ideal for disaster recovery and BCS?

"Absolutely! However, the consumer broadband systems now widely deployed in homes and governmental agencies are in need of a way to instantly block residential and commercial users during time of disaster need. Since it is unlikely that most disasters will affect all states simultaneously, a defined bandwidth and hub facilities could be shared by all users," says Whetstone. "Of course, additional resources should be available in case of a national emergency. Access to the resources must be as simple as pointing the dish and transmitting. DAMA systems can meet all of these requirements, if properly implemented."

D-TDMA To The Rescue

The worst case scenarios always have to be taken into account when BCS or disaster recovery blueprints are taking shape. After the primary network crashes, the back-up network has to light up immediately, and it not only has to be ready to move traffic on a priority basis, but it has to do so seamlessly.

The Reston, VA-based company, iDirect, offers a BCS solution based on its advanced broadband router technology known as the Netmodem, which incorporates its Deterministic-TDMA or D-TDMA technology. With the Netmodem, iDirect supports VoIP and data simultaneously using advanced queuing techniques. During catastrophic events, when network congestion can be excessive, iDirect’s D-TDMA technology offers a way to avoid the logjam.

SES Americom and Shared Data Networks are two companies that offer BCS using the iDirect solution. "This is all about a more effective approach to BCS in general, while giving users the ability to design their own networks based on their specific applications," says Sasmith Reddi, director of systems engineering at iDirect. "We can prioritize based on the application, whether it is voice, video or another application, and provide QoS either at the IP level or at the application level."

Oversubscribing the network and then providing guarantees is not rocket science. While the convergence of satellite technology and VoIP, for example, happened years ago, the team at iDirect is looking to go one step further with a Frequency Hopping (FH) solution that will add increased efficiencies to both the primary network and the back-up network using D-TDMA.

"As the number of remote sites grows, there is a need to create more efficient services, and to support multiple applications with carriers operating at different speeds. At the same time, this allows the remotes to use any carrier as demand dictates," says Reddi. "For example, video, voice and data applications from a disaster site can be efficiently supported on-demand using FH technology by enabling remotes to burst to carriers with available capacity."

Many scenarios must be addressed both from the standpoint of assigned bandwidth and rapid switching. Voice, video and data crisis models need to be examined with infinite flexibility in mind.

"Our remote terminal is a one-box solution that includes an IP router, satellite modem, QoS, link encryption and TCP acceleration," says Reddi. "Link encryption can be assigned to any or all remote sites over the same network using dynamic key exchange. This can be done whether the network in question is private IP, public IP or closed."

Providing The Right Space On The Ground

When it comes to BCS or disaster recovery, a well-equipped and well-connected workspace on the ground is essential. This is especially true when it comes to call centers. Hauppauge, NY-based Globecomm Systems Inc. and Agility Recovery–formerly GE Capital IT Solutions Disaster Recovery Services–have teamed up to offer this type of specialized back-up capability.

"The new contract supplements an existing contract that allowed businesses to recover in the event that a facility was destroyed or damaged by providing trailers with work stations and connecting by satellite through to our teleport for connection to company databases and the public switched telephone network," says David Hershberg, CEO of Globecomm Systems.

Large insurance companies, for example, are aware that Globecomm Systems Inc. and Agility Recovery recently demonstrated their capability to restore and recover a call center with almost 400 seats, including voice, automatic call distribution, interactive voice response and all application databases.

Dual Ku-band satellite links on GE 3 were engaged as Globecomm’s teleport beamed phone traffic to a pair of frame relay networks with one network dedicated to call center data, and the other carrying application data to an insurance company’s corporate headquarters. The aggregation, optimization and prioritization of all VoIP traffic went smoothly, due to Globecomm’s proprietary techniques.

"We are now providing automated call distribution software for operating a call center. The system we offer is a completely IP-based system with data and VoIP. This is very effective because the bandwidth is shared between voice and data and efficiently uses satellite bandwidth," Hershberg adds. "Full remote switching on site is built in as well."

According to Hershberg, this offering is very cost effective because the customer only pays a nominal amount to have it available and an additional cost if it has to be activated. "The system we announced is very sophisticated because it allows a customer to put back in service a call center for its customers in 24-hours. Other requirements we are seeing are for a complete overlay of a satellite network between enterprise and government locations," Hershberg says. "Over the past year, it was apparent that many more companies and the U.S. government are more interested in these services."

Satellite Solutions Alone Will Not Work

Like everyone else, the satellite industry is looking toward the next emergency or disaster with a shared sense that much has been done to make the job of infrastructure restoration and disaster recovery quicker and easier for both the public and private sectors.

As noted above, there are still gaps in terms of what constitutes the national blueprint, and the overall effectiveness of the planning to date gets mixed reviews. The challenges are enormous and the satellite industry by itself cannot be expected to provide all the answers.

By encouraging further dialog and cooperation, the satellite industry can help forge a more flexible and more effective response when the time comes. Given the dire fiscal condition of many states and local municipalities and the slow pace of federal funding, the satellite service providers and hardware vendors have to be patient, innovative and willing to work closely with decisionmakers at all levels to really make a difference in the long run.

Peter J. Brown is Via Satellite’s Senior Multimedia & Homeland Security Editor. He lives on Mount Desert Island, ME.