News about the long awaited Keynote Systems VoIP study is out, and no surprise AT&T CallVantage won for best quality, while surprisingly Vonage won for most reliable carrier.
Internet Telephony also has a viewpoint.
No study is ever perfect, but this is what we in marketing like to call a “stake in the ground.” In talking with the Keynote analyst I found a few areas of concern in the methodology that in talking with some carrier representatives under the cloak of secrecy over the weekend that make me feel that Keynote has done a super job, clearly the study was developed from only an IT centric standpoint of testing, and not with a Consumer Voice Services approach.
A) The use of T-3’s. That’s very big pipe and according to my sources the Telephone Adapters and network are optimized for DSL and Cable. While the T-3’s only represented a portion of the test, the fact that they were included skews some of the data.
B) Vonage and Skype rely on the public internet while Verizon’s VoiceWing, Packet8 and AT&T’s CallVantage uses a managed networks. Here apples and oranges are being compared as traffic over the Public Internet is never as good.
C) The use of Comcast in the Bay Area. This is one of the older TCI physical plant markets versus say Time Warner which in NYC is a more updated build out by comparison.
D) Companies like Broadvoice, VoicePulse and AOL should have been in the trial.
The above not withstanding, Keynote, which did this study on their own, has done VoIP consumers a great service. They now know what service sounds best. But, I think I told you that months ago back in December.
Along with that I have some other observations which may help temper the overreaction to the study in some quarters (i.e. mainstream media and blogosphere)
1) This study seems to have a more ISP centric bent and is not oriented around consumers as much, at least based on what was disclosed.
2) Without seeing the full report data it is hard to really assess what Keynote actually measured beyond basic connectivity, call initiation, ring through and voice quality
3) Most non-technical consumers (i.e. those already on PSTN) are looking for total value proposition…that would include the ability to port numbers, ease of installation and home wiring integration, a full mix of reliable features, service that mimics and surpasses their PSTN service, E911, price, etc.; this report ignores those key consumer and non-technical attributes
4) Consumers are better served by looking to their perceived trusted sources for tech insight like PC Magazine and others rather than relying on a very inside the tech world view that is more geared for those in the business, not those using the business.
Keynote also indicated to me that they will update the study and likely expand it in a few months with another round. Hopefully they take the constructive criticism the right way and make the next study more holistic and well rounded.