System update led to BlackBerry outage

System update led to BlackBerry outage

	Published : Sat, 21 Apr 2007 04:26 By : Agencies Print this Story

NEW YORK (AP) - After two days of silence about a lengthy outage in its BlackBerry e-mail service, the company that makes the addictive mobile device issued a jargon-laden update indicating that a minor software upgrade had crashed the system.

The statement Thursday night by Research in Motion Ltd. said the outage from Tuesday evening into Wednesday morning was triggered by 'the introduction of a new, non-critical system routine' designed to optimize the cache, or temporary memory, on the computer servers that run the BlackBerry network.

RIM said 'the pre-testing of the system routine proved to be insufficient.'

The failed upgrade apparently set off a domino effect of glitches, which the company referred to as 'a compounding series of interaction errors between the system's operational database and cache.'

The Canadian company said a 'failover process' to switch to a backup system 'did not fully perform to RIM's expectations.' That led to a delay in restoring service and 'processing the resulting message queue,' a reference to the backlog of undelivered e-mail that accumulated during the outage.

While most of the outage happened outside 'work' hours, the always-connected mentality fueled by BlackBerry's success left many users feeling disjointed and aggravated when their devices stopped buzzing. Grumbles were heard at the highest levels of business and government, including the White House and the Canadian Parliament.

The outage and the company's delayed, tightlipped response to the situation angered some customers. It is an approach RIM has taken with past service outages, which in fact have been rare.

Jim Balsillie, RIM's co-chief executive, downplayed the criticism of the company's communications as 'a trifle unfair,' because the focus was restoring service, and the primary means of contacting users was unavailable.

'The issue is just how do you tell people what it is when it is e-mail that people are counting on, and that very communications path is down,' Balsillie told The Associated Press in an interview.

Furthermore, he said, there was no information to disseminate until the cause was identified.

'Once we had the facts, we made sure they were available,' he said. 'People got a very accurate characterization of what it was.'

Yet with the company rapidly expanding beyond its longtime focus on business users -- the new BlackBerry Pearl has been a smash hit with consumers since its launch last summer -- some experts say RIM needs to get more savvy in dealing with problems.

'So far, all we have gotten from RIM are explanations fit for engineers, not customers,' said Richard S. Levick, whose firm Levick Strategic Communications LLC specializes in crisis communications.

During the last major failures nearly two years ago, RIM waited hours before confirming the problem, then issued a cryptic technological description of what happened.

This time around, from the time the e-mail ceased flowing Tuesday evening, it took RIM more than 12 hours to issue a vague three-sentence statement acknowledging the disruption. No further updates were provided until late Thursday's statement, prompting criticism in online forums and Web logs.

'They have to stop thinking like engineers and start thinking like a utility,' Levick said. 'When the telephone lines go down or the power goes out, the first thing these utilities do is try to fix the problem while simultaneously communicating with the media and customers. Why does RIM think it can't do two things at once?'

John Corcoran, a Tampa, Fla., franchisee in the Wireless Toyz retail chain, said his customers were generally patient.

'Because of how solid RIM has been in the past, BlackBerry users have been very understanding and willing to give them the benefit of the doubt this time around,' he said. 'Historically, BlackBerry users have been part of an exclusive, tech-savvy club. Because of the knowledge they hold it is very important for RIM to be upfront with them about any operational issues.'

RIM said it has ruled out security and capacity issues, along with hardware failure or core software issues, as the cause of the disruption. The company said it is improving its testing, monitoring and recovery processes to prevent such an outage from happening again.