Like most folks, I watched in some disbelief – and more than a little annoyance – at the 33-minute delay of the Super Bowl when a power surge knocked out half the stadium lights during the third quarter.
It was comical, surreal and fascinating how CBS tried to keep its audience engaged. It was also a big black eye for the most popular sporting league in the United States and the city of New Orleans as it was attempting to prove to America and the world that it was back from the disaster that was Hurricane Katrina.
Most disturbing was how such an event could occur at the biggest NFL game of the year, despite all the behind-the-scenes equipment testing and disaster prevention preps.
But it happened and became the biggest story for the papers, talk radio stations, cable news outlets and sports channels. Many of my business colleagues chuckled a bit at the situation when it occurred. I, on the other hand, saw the issue as a clear reminder to organizations that they need to prepare for worst case scenarios at all time.
If companies don’t think that a 30-odd minute unexpected network or power downtime isn’t a big deal, think again.
That half-hour-ish delay was one of the most awkward moments in sports, as network commentators tried anything to keep the conversation going, even though that meant engaging in the painful exercise of over speculating on how the interruption would affect the teams’ play. Even more painful was the fact that the power outage did stop the Ravens’ momentum and allowed San Francisco to get back in the game, thus making it a competitive match.
Companies need to take note here. What may seem like a short amount of time in many circles, your customers expect top performance from you 24 hours a day, seven days a week.
In their minds, there is no excuse for not performing well, and each minute that doesn’t live up to standards will be viewed as unacceptable. So before laughing off the Super Bowl blackout as a “once-in-a-lifetime” event, organizations need to do the following to ensure that they don’t have an unforeseeable occurrence at the worst possible time.
Quantify the impact of a service interruption. Take into account the financial, contractual and regulatory implications of experiencing an event. I’m sure the Super Bowl Committee and the Mercedes Superdome didn’t think the power outage was probable, but given the fact that the average rate for a 30-second ad on CBS during the game was $3.75 million, there was a lot of money lost in the span of 30 minutes for something that was deemed not very likely to occur.
Decide how much downtime is acceptable. Use the previous conclusions to guide a discussion of how much downtime is acceptable to your business. Keep in mind that for certain organizations, downtime can have an almost irreparable impact on the reputation of a brand. I’d put the Super Bowl in this category. Regardless, decision makers must understand the implications of such events, and make sure the appropriate steps are taken to maintain the necessary uptime.
Calculate your company’s recovery point objective (RPO) and recovery time objective (RTO). An RPO refers to the amount of lost time and, as it pertains to your IT systems, the data your company can afford to lose. The RTO focuses more on how long you can go without normal operations. For instance, some companies might be comfortable if their business were shut down for a few days, while other organizations – a la the NFL – would be in serious trouble if their mission-critical systems were down for more than an hour. Setting both an RPO and an RTO are essential to determining what kind of measures should be taking to ensure appropriate business continuity and enable quick recovery after a disaster.
So laugh as you may at the incredible hiccup that occurred during the biggest American Football game of the year, the incident should also serve as a warning for organizations to review their disaster recovery plan. There are some times when having a plan still isn’t good enough.
Alfrey is the managing director at redIT U.S., a provider of cloud services. He can be reached at firstname.lastname@example.org.