Is Your Data Center Prepared for a Natural Disaster?
Subscribe to vXchnge Blog
The threat of a natural disaster looms large in the minds of data center providers around the world. Managing the power, cooling, and security demands of a data center is difficult enough even before taking threats such as hurricanes, earthquakes, or flooding into consideration. Maintaining data access in the event of a disaster can mean the difference between a company’s success or failure. About 70% of companies have experienced data loss due to accident or disaster; even more troubling, 60% of those companies go out of business within six months of losing data due to disaster.
It’s imperative, then, that every data center has a comprehensive plan for protecting data in the event of a natural disaster. While every effort should be made to maintain server uptime, data centers must consider the possibility that they won’t be able to deliver on that promise in a disaster situation. After all, even the most robust SLA won’t amount to much if the power fails and no one can physically access the data center to get critical systems running again.
What Could Possibly Go Wrong?
In a word, everything. Here are just a few examples of natural disasters that left data centers reeling:
- Lightning: They say lightning doesn’t strike the same place twice, but in 2015 one of Google’s European data centers was struck by lightning not once, but four times, causing errors in 5% of the disks responsible for Google Compute Engine (GCE) instances. Although the company restored many of the drives, an estimated 0.000001% of data stored in the data center was irrecoverably lost. While that might not sound like much, try telling that to the customers who were affected by it.
- Hurricanes: According to National Geographic, 2017 was the most expensive hurricane season in U.S. history, costing roughly $200 billion. With their combination of high winds, storm surge, and heavy rains, hurricanes are one of the most dangerous natural disasters data centers must contend with. The sudden flooding resulting from Hurricane Sandy in 2012 caused extensive data center outages in New York and New Jersey. These failures were made even worse by the fact that backup systems were located in the same geographic region and where knocked out by the same weather event.
- Tornadoes: A devastating 2011 tornado ripped through several hospital buildings in Joplin, Missouri, one of which was a data center. While none of the data lost was mission-critical, that was only because most of the information stored there had been migrated to a new offsite data center just a few weeks earlier. Hospital officials noted that if the tornado had hit a month earlier, the data loss would have been catastrophic and rendered the hospital completely inoperable.
- Flooding: Severe flooding in Leeds, UK caused a Vodafone data center to temporarily lose power during Christmas of 2015. While data loss was negligible, the power outage disrupted mobile phone service temporarily. Vodafone, of course, has a bit of history with flooding, having suffered one of the most infamous data center disasters when its Istanbul data center was devastated by flooding in 2009.
- Earthquakes: So far, data centers have been lucky. Modern architectural standards and additional precautions (such as special enclosures and rollers for server racks) have gone a long way towards protecting data centers from earthquakes, even in high-risk areas.
- The Unexpected…: Disaster planning is all about expecting the unexpected. Take, for instance, the squirrel that knocked Yahoo’s Santa Clara data center offline for several hours in 2010, or the truck that drove into a transformer feeding power into a Backspace data center in 2007.
Is Your Data Center Prepared?
The very first step any data center should take to prepare for disaster is to perform a comprehensive risk assessment. This review will identify both the likelihood and expected consequences of potential disasters. Once these risks have been identified, the provider needs to create a step-by-step checklist detailing what actions need to be taken in the event of each specific disaster. All relevant staff and personnel should familiarize themselves with these plans and conduct drills on a regular basis to ensure everyone knows what to do in a disaster scenario.
Keeping the network up and running is a key consideration in addition to preserving customer data. Every moment of downtime carries with it real financial costs. Any good data center should already have extensive network redundancies incorporated into its computing infrastructure, but these backup plans need to be even more reliable in a disaster situation. Regular testing is essential to ensure that when power is disrupted and systems fail, customer data and critical operations are kept online and secure. Also, all relevant staff must be trained on what to do if the redundancy system doesn’t operate as designed. This normally involves critical manual intervention and personnel must be trained in a ‘hands on’ environment to gain the experience and understand of the process and system reaction to the manual transfers.
Data centers also need to consider how disasters will affect the infrastructure around them. When Hurricane Sandy struck New York City, for example, many backup generators failed because they ran out of fuel and could not be replenished due to the flooded city streets. Because physical access to the data center could be limited and remote access may not be possible, it’s vital to have automated backup systems in place to ensure that customers’ mission critical data and services are not lost or disrupted in the event of power failure.
If data is going to be backed up at another data center, that facility should be located far from the one being affected by a natural disaster. Data center providers with facilities spread across a wide geographic area are better able to ensure that a large scale natural disaster will not be able to take down all of their services.
As natural disasters like hurricanes and wildfires become more frequent, data centers must take active measures to protect their facilities and, by extension, their customers. Although cloud services make it easier to back up essential assets and modern construction techniques can better protect equipment from physical danger, data centers still rely upon factors outside their control to keep up and running. Only by preparing to deal with a disruption to their everyday operations can they truly be ready for whatever disaster nature decides to throw at them.
Is your data center prepared?
About Ross Warrington
Ross is a Regional Vice President, Operations at vXchnge and is responsible for managing all 14 data center locations. With more than 30 years of experience, Ross has managed data center construction, engineering, repair and maintenance, leading him to the emerging business of colocation. Ross has participated in colocation design, construction, engineering and operations in his various roles. He is a Vietnam Era Veteran, having served four years in the United States Marine Corp from 1969 to 1973.