Amazon Web Services: Outage During Bid for CIA Contract

Amazon’s Web Services went down on Sunday due to a technical issue at a North Virginia data center. The outage was caused by a problem with a single networking device, and reveals that many companies do not distribute their Web services in different locations for service redundancy. This comes as Amazon is bidding on a CIA contract to manage their data services, and competitors are critical of whether Amazon can manage the demands of government data.

“Amazon blamed the outage on glitches with a single networking device — what it called a ‘grey partial failure’ that resulted in data loss — and said it was conducting a forensic analysis,” explains Businessweek. “The entire incident lasted all of 49 minutes, but like many recent cloud service outages, the resulting questions are likely to last considerably longer.”

Amazon and other cloud service providers recommend geographic redundancy with services spread across many locations preventing any possible service disruptions. The outage reveals that online services such as Instagram, Airbnb, Vine and Flipboard were all affected.

“The biggest surprise from Sunday’s outage may be who was affected. Instagram is now owned by Facebook, which has invested considerable resources to create its own global data network,” explains Businessweek. “Vine is owned by Twitter, which has similarly earned its own hard-won infrastructure expertise. Likely both have some degree of redundancy, but it clearly wasn’t enough. And apparently Facebook and Twitter have yet to bring their recent acquisitions onto their own computing networks.”

“Inside a single facility, there are simply too many ways to shoot one’s own foot,” said James Hamilton, an Amazon engineer. While he recommends redundancy, Hamilton also accepts that “with incredible redundancy, comes incredible cost,” in that a truly distributed model is expensive.

The outage occurs as Amazon is rebidding on a $600 million CIA hosting management contract from last year. IBM, a competing bidder, challenged Amazon, claiming that the spy agency did not properly evaluate their bid, reports The Seattle Times.

Some competitors claim that Amazon Web Services (AWS) are not capable for handling the demands of managing the sensitive data of government agencies, although AWS has provided services for the NASA Jet Propulsion Laboratory and Obama for America. However, if the CIA can rely on Amazon, then other companies may follow to Amazon as well.

The contract would allow Amazon to manage a private data center owned by the CIA, rather than an AWS owned data center. The agency will require a level of security that AWS cannot provide, but Amazon is willing to change its approach to security to win the contract.

“The stakes are particularly high for AWS. Just as winning the contract was a point of validation for a service, losing a contract it had once won could undermine the perception that its services are capable enough for the most demanding customers,” notes The Seattle Times.