Twitter-storm: Blaine leaves, blame flies

by Mathew on April 23, 2008 · 32 comments

Update 2:

More turmoil at Twitter: Lee Mighdoll, the engineering VP that Twitter just hired in January (!) has left the company as well.

Update:

My friend Mike McDerment of FreshBooks interviewed Blaine during his road-trip to the Future of Web Apps conference, and the video is here. And if it’s video you’re interested in, Mike Arrington defends his post in a video comment that he posted on TechCrunch using the new Seesmic video-comment feature. He says Blaine “owns” the failure as chief architect, and should have owned up to it instead of getting his friends to attack TechCrunch and other critics.

Original post:

If you need any proof that Twitter is the hot topic in the blogosphere at the moment, all you have to do is look at the volume of blog posts about the service — and related services such as Tweetscan and Twist (Twitter trends) — on sites like TechCrunch. The latest brushfire has to do with the departure of Blaine Cook as the outage-plagued service’s chief architect. Mike Arrington says that it’s obvious that he failed to scale the service properly, and therefore it’s a good thing he’s gone (and has apparently been replaced by a couple of new hires).

As support for this argument, Mike uses a presentation that Blaine made at a conference last year, in which he claimed that scaling applications that use Ruby on Rails is “easy” and suggested that Twitter’s problems were mostly behind it. As anyone who has been using the service much over the past year knows, that statement was… well, overly optimistic. Is that Blaine Cook’s fault or is it that Rails doesn’t scale? There’s no question that Twitter usage has skyrocketed over the past six months or so. For his part, Blaine says the departure was amicable and that it was just time for him to move on.

There are plenty of comments from supporters (ironically, using Twitter) who say that the former Twitter architect is a class act and a great software engineer, including Tom Coates and from Tara Hunt. Others seem to support Mike’s case that if you make a public claim that your software is easy to scale and then you fail to do that consistently, then you deserve to be held to account for that. In any case, it’s more than a little ironic that while this blogstorm is swirling, Blaine is presenting at the Web 2.0 conference about how to scale a Web application.

  • http://nimble.tv MMG

    lol…great closing sentence.

  • http://www.mathewingram.com/work mathewi

    Thanks, MMG :-)

  • Pingback: Interview with Blaine from Twitter

  • http://InternetDuctTape.com engtech

    Ugh.

    What a reason not to work at a web startup… I don't think any programmer wants this kind of publicity. :)

  • http://freepository.com John Minnihan

    Hey 'thewie (can I call you that?),

    What I've read about Blaine is that he's a smart & decent guy. So, keeping him out of this personally, it's fair to pin the *responsibility* of the architecture's performance & scalability on the architect.

    Rails' scalability has been in question for a long time (see http://commavee.com/2007/11/16/twiiter-outage/ ), so though Blaine is getting a serious thumping that seems a bit personal, decisions that affect a service's stability have to be justifiable.

    Hindsight is 20-20, no doubt. I want to be clear here – as a founder & Chief Architect – that mistakes occur. You will make the wrong choice at some point. You just have to be able to quickly & effectively correct the mistakes.

    Usually, the architect makes the correction. In this case, it may well (I have no idea) have been Biz & Ev that made the correction.

    Then again, perhaps we're all full of it & Blaine really did just resign.

  • http://InternetDuctTape.com engtech

    Doesn't that TechCrunch article seem more fitting for ValleyWag?

  • http://www.mathewingram.com/work mathewi

    I guess it's hard to know for sure, John — maybe Ev and Biz decided
    they needed more heft in the architect department, and Blaine was
    ready to move on, and the two events came together. Thanks for the
    comment.

    And no one calls me 'thewie except Stuart :-)

  • http://www.mathewingram.com/work mathewi

    Actually, it seems a little mean even for Valleywag :-)

  • http://freepository.com John Minnihan

    I stand corrected – Mathew. ;)

  • http://www.scripting.com/ dave

    Congrats Matt, you got this one right. It's impossible from the outside to tell who's responsible for the last outage, but Twitter was pretty reliable for the last few months, so that speaks well for Cook's work. However, the fact that this guy left two weeks ago might explain why it took so long to clear the outage.

  • http://joeduck.wordpress.com JoeDuck

    it’s more than a little ironic that while this blogstorm is swirling, Blaine is presenting at the Web 2.0 conference about how to scale a Web application.

    Ha – another sweet saga of internet irony!

    I think one may need to factor .. money … into this story. Twitter is wildly successful in terms of traffic and adoption, but it's not monetizing that success. Scaling up to an extremely robust infrastructure could be throwing good money after … more of no money. Many bubble companies developed huge and robust architectures to handle trivial traffic. I'm not at all convinced Twitter is wrong to set their priorities as they appear to have done – a great service with a second class infrastructure until they figure out how to turn a buck from all the Twitterers.

  • http://corvusconsulting.ca Todd Sieling

    I thought the TechCrunch post was scapegoating. To say that Twitter has failed to scale without posting any kind of uptime numbers is bad enough, but the overall perception of Twitter's reliability seems skewed.

    Look at Twitter's performance at SxSW 07, where it really got noticed and started to take off. Any service would have its hands full keeping up with the sudden attention and growth, but Twitter fell down a lot at that event. Forward to '08, where there are nearly double the number of attendees, they all have iPhones now, and Twitter is very much on their minds and fingertips from the start. During that frenzy of use, I heard about only momentary failures.

    Throughout that year, Twitter opened itself up through an API, adding scores of other web applications and their users to the demand, and yes uptime has gotten better. Somehow this all adds up to an inability to scale?

    Since moving to a different host up to last weekend, a week or so after Blaine left, the service had been quite reliable, and I think that makes the weekend outage that much more painful. It hurt more because it hadn't hurt in a long while. I can understand people freaking out a bit, but to write such a scathing smear of Blaine's role there looks terrible on TechCrunch.

    As for asking friends to defend him, I think the grownups in the room learned to walk by that kind of baiting in highschool.

  • http://www.mathewingram.com/work mathewi

    Thos are fair points I think, Todd — it's true that Twitter made it
    through SxSW without even a hiccup, and maybe that got people thinking
    the worst of the stability problems were gone. And even the issues
    that have occurred since then haven't been all that big a deal really,
    although they have been irritating. And there's no question they have
    had to grow at a huge rate, given the number of other apps that are
    being built on their API.

  • http://burningbird.net Shelley

    Arrington's post is frankly obscene, as well as nothing more than cheap link bait. He wouldn't know technology if it bit him in the butt.

    I don't know Blaine, but he left a couple of weeks ago. The Twitter folks also came out and said they tried a new caching scheme that sounded like it failed. Considering how much use this application is getting, I'm rather amazed it doesn't fail more often. Even Google's apps, as well as Amazon's have been known to fail, and no one accuses these two companies of not knowing how to scale.

    Don't you guys ever write about anything useful anymore? All I read now is whining, blame games, petty bickering — good lord, you guys are a piece of work.

    I'm not going to write at Techcrunch, because frankly, that site has become a dead bore. But you know Matthew, you used to write about decent stuff once upon a time. Now it seems like all you're doing is going for the attention points.

    PS I was an architect at a start up once. I bet there isn't one true tech person who is playing this same blame game. No, it's all of you who haven't a clue.

  • http://avc.blogs.com fredwilson

    this is so upsetting. blaine's a good guy. he was a key talent behind building a great web app. he's moving on to do other interesting stuff and twitter's bringing in people with different skills that are more appropriate at this time.

    and somehow blaine gets pummeled in the process. it's so wrong. if there's accountability to be had, it should be on the CEO and the board, not on a great developer.

    fred

  • http://www.mathewingram.com/work mathewi

    I agree, Fred. It's true that Blaine was the chief architect, so the
    buck stopped with him in a lot of ways — but it still seems unfair to
    pin everything on him, especially when Twitter has done a pretty good
    job of handling an incredible growth curve.

  • http://www.gilhildebrand.com/afterthought giltotherescue

    although i'd like to stick up for another architect, twitter is a really simple web service. it succeeded because it's a great idea, not because of brilliant engineering.

    the reasons for twitter's scaling problems are irrelevant. an architect's main job responsibility is to design a solid, stable infrastructure, and i don't have a problem holding him accountable for a failure in that department.

    all that said, there's no reason this should be such a loud conversation. i haven't read the arrington post referenced above and don't plan on it.

  • cvg

    Money is less of a problem than you think. Not only does the company have access to a good amount of capital, but they are likely getting a bit of revenue share from wireless providers. Once a company passes a certain threshold of SMS usage, they get a kickback from wireless providers for driving $.20 per message traffic.

  • http://www.mathewingram.com/work mathewi

    That's interesting, cvg — I didn't know that.

  • http://www.howardlindzon.com howardlindzon

    why doesnt twitter shut off arrington or chrage him an obscene amount to use the such terrible service. the universe would be a little better. an amazing giant putz is arrington. Shame on twitter for not shutting him down and making him use edgeio to spread his word

  • inger omenu

    blaine is a class act. i can't believe you let arrington's callous and off-the-cuff attack pass without comment. but such is the case in today's echo chamber: one blog reposts the natterings of another who reposts the natterings of another without any idea of the real facts. the sliming of blaine is so patently unfair and innacurate that it makes me tremble with rage.

  • http://joeduck.wordpress.com JoeDuck

    Thx cvg – didn't know about the SMS cash aspect. However the capital they have access to is still subject to what I'm talking about – ie they are well-advised to scale expenses to be somewhat in line with cash flow, which I'm still confident is minimal at Twitter, not to mention Facebook and other social enterprises.

  • http://joeduck.wordpress.com JoeDuck

    … making him use edgeio to spread his word

    Ouch!

  • http://www.howardlindzon.com howardlindzon

    He dishes it out. He can take it.

  • http://joeduck.wordpress.com JoeDuck

    one blog reposts the natterings of another who reposts the natterings of another without any idea of the real facts

    An excellent point. If it gets any worse we'll all be absorbed into network TV news

  • http://www.mathewingram.com/work mathewi

    good one, Joe :-)

  • http://joeduck.wordpress.com JoeDuck

    Thx cvg – didn't know about the SMS cash aspect. However the capital they have access to is still subject to what I'm talking about – ie they are well-advised to scale expenses to be somewhat in line with cash flow, which I'm still confident is minimal at Twitter, not to mention Facebook and other social enterprises.

  • http://joeduck.wordpress.com JoeDuck

    … making him use edgeio to spread his word

    Ouch!

  • http://www.howardlindzon.com howardlindzon

    He dishes it out. He can take it.

  • http://joeduck.wordpress.com JoeDuck

    one blog reposts the natterings of another who reposts the natterings of another without any idea of the real facts

    An excellent point. If it gets any worse we'll all be absorbed into network TV news

  • http://www.mathewingram.com/work mathewi

    good one, Joe :-)

  • Pingback: Deals: @Twitter finally buys @Summize » mathewingram.com/work |

Older post:

Newer post: