Recovering from Disaster
2/20/2013
Making things better, not worse, after the unexpected
Sandy, Nemo, and several other major storms have brought quite a bit of the unexpected to us the last year. And when uncertainty strikes, we’re reminded of what the Shubert IT department focuses on day in and day out: it is critical to minimize any system outages during any disaster that may occur.
We can’t avoid natural disasters, but it is essential to have a plan in place for immediate and long term recovery.
Multiple Data Centers = Limited Service Interruption
Shubert Ticketing’s primary data center is in Hackensack, New Jersey, where the ticketing system, websites, and ancillary systems reside. In the past we operated a backup data center in our offices in Manhattan. This provided us with system redundancy, including supporting our entire business for six weeks in the summer of 2011.
In October we established a new backup data center at a collocation hosting facility (COLO) in Edison, New Jersey. With multiple generators and dual power feeds from different sources, this COLO offers our system a more robust backup than Shubert’s New York office. Several communications providers, with entry to the facility at two sides of the building from different feeds, add to the heavily redundant backup system. All of this minimizes the chances that power or communications will go down, even when a major storm like Sandy hits.
How does it work? Every transaction and system update (software, content, inventory etc.) that takes place in Hackensack is replicated real-time to both a backup system in Hackensack and to the COLO facility.
If the main data center in Hackensack ever goes offline for any reason, all of the orders, files, customer records, and systems would be available from the COLO, with no loss of data or information. Regular Disaster Recovery Tests
Every quarter we test our ability to switch from our main data center to the COLO. These are called Disaster Recovery tests. These tests ensure that all systems and services are working as expected.
Over the course of two hours (during off-peak sales times), we move all of our operations (ticketing system, websites, etc.) to the COLO facility, replicating back to Hackensack. Box office machines, call center machines, web transactions, credit card authorizations, and other services are all tested to make sure everything is functioning smoothly, and that no data gets lost.
We admit that disaster preparedness isn’t sexy, but it’s certainly better looking than lost orders. And while no one can know what unusual events Mother Nature (or our power grid) may send our way, we’re confident that we’re ready for it.
Making things better, not worse, after the unexpected
Sandy, Nemo, and several other major storms have brought quite a bit of the unexpected to us the last year. And when uncertainty strikes, we’re reminded of what the Shubert IT department focuses on day in and day out: it is critical to minimize any system outages during any disaster that may occur.
We can’t avoid natural disasters, but it is essential to have a plan in place for immediate and long term recovery.
Multiple Data Centers = Limited Service Interruption
Shubert Ticketing’s primary data center is in Hackensack, New Jersey, where the ticketing system, websites, and ancillary systems reside. In the past we operated a backup data center in our offices in Manhattan. This provided us with system redundancy, including supporting our entire business for six weeks in the summer of 2011.
In October we established a new backup data center at a collocation hosting facility (COLO) in Edison, New Jersey. With multiple generators and dual power feeds from different sources, this COLO offers our system a more robust backup than Shubert’s New York office. Several communications providers, with entry to the facility at two sides of the building from different feeds, add to the heavily redundant backup system. All of this minimizes the chances that power or communications will go down, even when a major storm like Sandy hits.
How does it work? Every transaction and system update (software, content, inventory etc.) that takes place in Hackensack is replicated real-time to both a backup system in Hackensack and to the COLO facility.
If the main data center in Hackensack ever goes offline for any reason, all of the orders, files, customer records, and systems would be available from the COLO, with no loss of data or information. Regular Disaster Recovery Tests
Every quarter we test our ability to switch from our main data center to the COLO. These are called Disaster Recovery tests. These tests ensure that all systems and services are working as expected.
Over the course of two hours (during off-peak sales times), we move all of our operations (ticketing system, websites, etc.) to the COLO facility, replicating back to Hackensack. Box office machines, call center machines, web transactions, credit card authorizations, and other services are all tested to make sure everything is functioning smoothly, and that no data gets lost.
We admit that disaster preparedness isn’t sexy, but it’s certainly better looking than lost orders. And while no one can know what unusual events Mother Nature (or our power grid) may send our way, we’re confident that we’re ready for it.