8 x 8 Has an Outage

I received an email late yesterday afternoon from a reader about 8 x 8 suffering an outage. Unlike Tom Keating, who posted, I didn’t post, and was relieved that their PR folks had been quickly in touch with their customers over night, explaining the situation.

In an era where no company (other than maybe the Bell Operating Companies) have everything in one NOC that does everything, this kind of stuff happens. It just seems to happen that 8 x 8 gets put under the microscope even more than Vonage whenever an outage occurs. Maybe, it’s because they were one of the first into VoIP, or maybe it’s because, like Vonage, they’re a public company.

My instincts tell me there’s more to this story than we know from their very clear notification, sent to customers:

Dear 8×8 Customer,

Today we experienced a disruption in our services that impacted your voice service. The outage appears to have been caused by a data failure with one of our underlying carriers, and we are working on identifying the root cause and why our backup services did not failover as they are designed to do.

Service has since been restored to most of our customers, and we are in the process of fully restoring service to all customers. If you are currently without service, please unplug the power connection to your 8×8 equipment, wait a few seconds, and then reconnect the power connection.

We understand the importance of providing reliable phone service to you and deeply regret the interruption this issue has caused. Delivering on and exceeding the service level goals we set for ourselves is our top priority.

We value your business and thank you for your patience.

Dan Weirich

President, 8×8, Inc. (Nasdaq: EGHT)

The reason I say this, is having worked with enough of the success stories in VoIP over the past few years, like GrandCentral (acquired by Google), HiDef Conferencing (acquired by Citrix Online), Web Dialogs (acquired by IBM), Sightspeed (acquired by Logitech) and others, one thing I’ve learned is that the outage isn’t usually the fault of the brand you’re buying from.

Many times the fault lies with the data center, an underlying carrier or a third party who supplies some piece of software that gets updated but wasn’t put into a test network first. What’s really bad though is since so many companies now use so called “standards based” software, when one goes down, they can sometimes all go down. I remember when this happened with DSL across the nation. One supplier at a time, over a series of weeks, simply because they all had the same middleware suppliers, and as the updates rolled out, so went the service outages. It just happens.

While some may be shocked, I’m actually very happy that 8 x 8 has been so responsive, both publicly and privately to their customers and to the media that covers them, like Tom. In our Twitterfied era of no more lag time, rapid response to what could be a crisis, means crisis averted. But in the communications era that means even shorter windows of time exist, but that doesn’t mean the media should jump to conclusions. Tom was very fair in his post, but I have seen others at times far less forgiving, and who won’t even wait or give the affected company a chance to learn what’s wrong, let alone be able to comment. This isn’t always “Spin Control” but often times deals with a desire to be accurate, and when possible, reveal exactly what happen, and even on rare occasions, assign who was at fault, while still taking the blame.

Brands that own the customer, and which are public facing are of course who has to shoulder the blame. But often times, they’re just the face, while some supplier is the real culprit. In the case of this outage, I would expect that to be the case. Hats off to Dan Weirich for being clear as to what happened, and yet not outing just who it was. No one would win there with that approach.

This same approach applies to a responsible press. If you promise updates to your coverage, and one is provided, then it’s important that as a responsible media that you do just that. It’s one thing to “break a story” that’s breaking news, but a responsible press has to continue to monitor and update, not simple grab a headline, and leave the good news off the page. That’s tabloid journalism, not a responsible media. So when the PR team does their job, the press has to do theirs too.