While Twitter's infamous "Fail Whale" was a hallmark of the platform's early days, VP of Platform Raffi Krikorian said onstage at Structure 2014 in San Francisco that by and large, the animal that signaled trouble for the company is rarely seen these days. That's because the company has developed a mature infrastructure that allows for prioritizing certain features when things get overloaded.
"When something happens, what are things we can automatically shut down?" Krikorian said. "What are things we can automatically degrade so we can make sure this core experience is working?"
Krikorian said that the company continues to face difficult challenges, particularly in managing the speed of real-time and also maintaining control when the firehose of tweets gets overloaded (say, during a particularly heated match at the World Cup). While the company experimented with Cassandra and MySQL, he said that the company is in the process of moving all data to Manhattan — which is what the engineers at Twitter prefer.
"I think my job is to provide infrastructures that nobody else needs to think about," Krikorian said.
While Twitter as a platform has become more efficient and scalable, Krikorian said that challenges remain in getting ever closer to real-time and managing bursts of activity. As tweets flow through the system in just 100 milliseconds, keeping up automatically is key.
"We don't print money, so I want to make sure that every single CPU is used as much as we possibly can, while still providing headroom for bursts."
A video embed of the session follows below:
Photo by Jakub Mosur
Related research and analysis from Gigaom Research:
Subscriber content. Sign up for a free trial.
