Outage gives BlackBerry a black eye

The second lengthy BlackBerry outage in less than a year has one consulting company advising clients not to rely solely on the popular smart phones for critical e-mail messages.
Enterprises that really need e-mail responses in less than eight hours to run their businesses should have a backup for the BlackBerry, Gartner analyst Ken Dulaney said Tuesday, in the wake of a crash that lasted about three hours on Monday. They should also set up an independent system to notify them whether an important contact has received or replied to an e-mail message, Dulaney said.

Late Tuesday, RIM said the outage was caused by a problem with an internal data routing system in the BlackBerry service infrastructure that had recently been upgraded. That upgrade was part of an ongoing expansion of the network, and this type of change had been done previously with no problems, it said.

Monday's failure followed a longer outage in April 2007 that RIM attributed to a problem with a minor software upgrade and a subsequent glitch in a failover process.

There are about 12 million BlackBerry users worldwide, on a variety of mobile carriers. The crash Monday affected users throughout the U.S. and Canada, preventing them from sending or receiving e-mail or carrying out some other functions, though no messages were permanently lost, according to RIM.

Gartner's Dulaney had harsh words for RIM, saying he had given the company a break after last year's failure.

"This does not appear to be a mission-critical system with the highest service-level agreements, and therefore, to entrust ... that a message will get to the end point is being a bit foolhardy," Dulaney said. He advises enterprises that are worried about dependability to ask RIM for details of its backup system and make their own decisions.

After last year's outage, RIM said it would enhance aspects of its testing, monitoring and recovery processes.

"RIM has made significant investments to improve its system recovery infrastructure and processes over the last year, which enabled service levels to return to normal quickly," the company said Tuesday.

"They obviously didn't do as deep a dive as they needed to on disaster recovery issues," Dulaney said. But he acknowledged the failure could have been caused by something completely unexpected.

The BlackBerry, which RIM introduced in 1999, quickly became the most popular tool for mobile "push" e-mail that automatically comes to a portable device with a wireless network connection. All current BlackBerry models are also phones, and the company now offers a wide variety of handsets with consumer-oriented features such as cameras, Web browsers and social-networking capabilities. But to get to and from the handhelds, e-mail messages traverse a complex system involving mobile operator networks, RIM's network operation centers and BlackBerry Enterprise Servers within companies.

Adding consumer features and keeping up with rapid customer growth, such as adding 1.65 million users in the quarter ended last Dec. 1, have probably made it harder for RIM to keep its network running smoothly, said Albert Lin, an analyst at investment bank Sooner Cap. But users accept some level of risk if they want such complex services at reasonable prices, he said. The only truly reliable system is the public switched telephone network, and it has limited functionality, Lin said.

"When it comes to reliable push e-mail ... it's still hard to find a solution that works better than BlackBerry," Lin said.

Gartner's Dulaney acknowledged that competitors, such as Visto and Good Technology (now owned by Motorola), are probably not significantly more dependable.

It wouldn't be impossible to make the RIM network more reliable, the analysts said.

"They have the technical capability to spend more and make it more reliable, but is that what their owners and shareholders really want?" Lin said.

Google, for example, has a more highly distributed network, but that would take money and time, and many customers wouldn't want to pay for it, Dulaney said. In the future, RIM may choose to create a higher-end service with greater service assurances and a higher price, he said.