A flood that damages mission-critical equipment is among the worst scenarios a data center can face, offering little hope for a quick fix. On April 13, 2015, the Zunesis team received a “first responders call” from a long-term public sector customer who had the misfortune to experience a broken water main that left their main data center under 2 feet of standing water.
The call for help came at 4 p.m. on a Thursday afternoon. Zunesis joined forces with HP and immediately responded to assess the damage. By the next morning, it was clear that the data center was living on borrowed time. With 18 racks of equipment compromised, it was imperative that the team immediately instigate a full Data center fail-over. Luckily, a secondary data center had just been completed, along with plans started for a full DR site; but those plans were stalled as the organization waited for additional funding to become available.
A two-fold plan was quickly put into place to swiftly expand the new data center to capacity while keeping the damaged data center up and running. The insurance company was immediately engaged to begin the replacement exercise while Zunesis and HP jumped in to “keep the lights on” in the existing data center. Immediately needed were a large supply of extra drives; power supplies; cables; switches; and, most daunting, an emergency 120TB SAN array to move data to safety.
By Friday morning, emails had been circulated to the highest levels of HPE and Zunesis requesting emergency assistance. Zunesis quickly moved a 3Par 7200 array from its own internal lab to the customer site and expanded an additional 100TB in an emergency drive order to migrate data. HP called in emergency supplies from all over the Americas. Extra drives, power supplies, cables, switches, and even servers arrived throughout the weekend. Within 48 hours, the teams had built up enough reserves to keep the data center live until all of the data could be migrated.
Although all IT organizations know that it’s critical to have a solid DR plan and failover Data center in place, the reality is that the Data center spotlight is on monitoring and compliance; and the loss of a main data center is crippling. The second most common cause of catastrophic failures (after electrical) in the Data center is water leaks. Taking the following three steps can help avoid a flood disaster:
Click to find out more about our Disaster Recovery Assessments.