8×8 Outage Wasn’t Their Fault

As I wrote yesterday, the outage at 8×8 wasn’t likely their fault so i was quite pleased when their PR person, Joan Citelli, sent me this note today:

Andy –

I wanted to provide you with an update/explanation of what caused 8×8’s service outage this past Tuesday.

In order to provide services to our customers from our three, redundant data centers, 8×8 utilizes several, non-contiguous Internet Protocol (IP) address blocks that are provided to us by Tier-1 Internet Service Providers (ISPs). Our service outage on January 12, 2010 was caused when an unaffiliated Tier-1 ISP interfered with the proper routing of all of these address blocks. While we were able to immediately restore service for some customers by broadcasting correct routing information via our backup data providers, the interference from the unaffiliated ISP continued for approximately four hours.

All 8×8 core software and networks continued to operate normally during the outage, but some customers were unable to reach our services. We are in the process of planning and implementing additional mechanisms to protect our customers from any similar routing interference in the future. We are also creating alternative information and communication mechanisms to more quickly communicate outage status and estimated time of repair information to our customers via our website, call centers and other communication channels should a similar issue affect our customers in the future.

Feel free to update your blog with this information.

Best wishes-

Joan

Any bets on whom the offending culprits were? Level3 has been a supplier to most VoIP companies including 8×8 and most likely AT&T requested control of the IP address blocks as they are doing massive network work these days. Given how both companies play the media relations game, I would be surprised if we ever heard an “mea culpa.”