Twitter and the importance of architecture

by Mathew on June 1, 2008 · 28 comments

The Twitter guys have been getting a lot of flak over the past few months (and rightly so, in many cases) for the unreliability of their app. But I think they should get some props for opening up and talking about what’s going on over there. Granted, this newfound desire to engage in dialogue (or damage control) should have come a lot earlier, at least in my opinion, but at least they are doing it now. They’ve even managed to foil Mike Arrington’s attempt to start a late-weekend bitchmeme by asking some rather pointed questions of the company.

Although they kind of dance around the specifics a little in their responses, Jack Dorsey and Biz Stone of Twitter seem to more or less confirm many of Mike’s suspicions about their architecture — they admit, for example, that the company is restructuring the way that the system functions, and have brought in some big brains in order to do so:

“We are currently taking a new approach to the way Twitter functions technically with the help of a recently enhanced staff of amazing systems engineers formerly of Google, IBM, and other high-profile technology companies added to our core team.”

I’m not a database guy or a system analyst, but it seems pretty obvious that the structure Twitter has isn’t capable of keeping up with the company’s growth, and may not have been designed properly in the first place to support what has become a gigantic multi-user chat service. In one of the posts on the Twitter dev blog, the company effectively admitted as much. Is that former chief technologist Blaine Cook’s fault? Who knows. But it has to change, and quickly, or Twitter could wind up losing much of the momentum is has gained so far (Michael Krigsman has some good advice on how to proceed).

I was talking with a friend of mine (who is a systems and database expert) about Twitter on the weekend, and he made the point that while startups shouldn’t try to forecast all the ways in which people might use their service — a point that Daniel Burka of Digg made during his recent design presentation at meshU — they still need to pick the right architecture early, or they will be in for a world of pain. In other words, remaining flexible is a good way to approach user-interface design, but not necessarily a good way to approach the design of the actual system structure that makes your service work.

  • Pingback: Twitter and the importance of architecture : PeapodStudios

  • BrianSullivan

    Lucid as always.

  • mathewi

    Thanks, Brian :-)

  • CyndyA

    Your friend has it exactly right. There's this whole argument that Web 2.0 moves so fast an the result is that companies HAVE to slap something together quickly. The problem is, many of these companies also end up going under just as quickly because they can't handle load. It's all well and good to have something that works great with a few thousand users, but if you want to succeed, you need to think past that. Socialthing is also struggling with this same problem. How many more companies are going to find themselves in this position before they think about getting an enterprise architect to help them design something in the beginning?

  • mathewi

    i think you're right, Cyndy — and you had a good post recently at
    Profy along the same lines. But I guess the tough part for companies
    like Twitter is, what happens if the way people use your app doesn't
    jibe with the way that you designed the architecture? In other words,
    whose fault is it if you design a system to do content management and
    people wind up using it as a global messaging system? Maybe it's no
    one's fault. As usual, I blame Scoble :-)

  • CyndyA

    Mathew, they screwed up at the get-go. It's original intent was group sharing of SMS messages. The second you think SMS, you have to assume messaging system. SMS IS messaging! I'm not even convinced that if people hadn't started using it the way they do, it would have held under an increased load. At the point at which you add an accessible API, you'd better have a backbone there. So this complaining that they didn't DESIGN it to be used this way is silly. They let it happen. It's like building a house on stilts on an eroding shoreline and then acting confuse when it gets knocked over in a huge storm.

  • mikepk

    Well I'm going to disagree with you here. As a startup there's only so much energy you have, and you must apportion your resources carefully. The truth is, we like to talk about scaling, but without steady growth and something people find compelling, all the scaling in the world won't help you.

    We significantly over-engineered our architecture at Grazr, spending lots of energy building a powerful and flexible system. We can potentially handle very large streams of data with huge numbers of users, *but* we're not seeing the user adoption that we'd been hoping for. We do many of the things that FriendFeed does, and if we'd been a little looser we could have launched them last October. I even gave a talk at MySQL about this, while it's sexy to work on scaling infrastructure, too much emphasis is a mistake.

    You're seeing now that, even with all their troubles, people are still loyal to Twitter and talking about it. It's definitely painful to change their architecture now, but they have the user momentum to carry them through these tough times. In the balance between building a scalable system, and just getting the users, it's a balance but the latter is more important.

    We've had almost no downtime, have ridden out all of our traffic spikes (from TechCrunch and others) with almost no problems. But no one talks about Grazr with the same passion and loyalty they do Twitter.

    Unless Twitter totally implodes, they'll weather this current pain and the majority of their userbase will stick with them (that's at least my contention). The fact that there are alternatives (and good ones with better stability track records) that have not gained traction should be an interesting indicator of what people care about.

  • mathewi

    That's an excellent point, Mike. If a startup had to choose between
    your problems (not enough users) and Twitter's (passionate users but
    the wrong architecture), I assume that most would probably choose the
    latter. I guess the hard part is that not many companies get to choose
    – they have to anticipate, which is always going to be inaccurate in
    a variety of ways. But your point is a good one: don't spend too much
    time on architecture, or you could wind up with a beautifully built
    and sturdy structure, but no users.

  • Pingback: Scobleizer — Tech geek blogger » Blog Archive The best comment on Twitter and architecture I’ve seen «

  • Pingback: Scoble’s post on Michael Kowalchik’s comment

  • curiousyellow

    I'm with MikePK (and Scoble). It's easy to criticize Twitter with hindsight, but how did they know how many users and tweets they would have to deal with? More importantly, Twitter, like Grazr, has now written a new chapter in start-up history. Twitter will become legendary for its scaling problems, and for its openness about dealing with these problems.

  • Pingback: » Blog Archive » Scoble Feedback

  • CyndyA

    Grazr broke the cardinal rule of Web 2.0: charging a fee. The free account is limited with the one stream and the paid accounts are $10 a month: steep in this economy for facilitating widespread adoption. It has nothing to do with who got out of the gate first; it has to do with which one was free. It's like comparing free apples with $1 a piece oranges. Everyone is going to take an apple even if they don't like them better.

    As for loyal users, how loyal are they? It would be interesting to see if my experience is the same as others; I'm spending far more time leaving comments on blog posts about Twitter than I am actually using the service. I'm very reliant on the IM portion of Twitter to actually use it, an even Twhirl, my second choice, horks every 2 seconds as “over limit.” Which means my use is steadily declining. If some one came along with a true alternative that made it easy to port data, I'm wondering how far that loyalty would go.

  • mikepk

    I see your point, but we already have a lot of free services that we've been offering. Hosting reading lists, the ability to create free feed widgets, and several others, that when it came to releasing streams we decided to try and charge for something. All the other services only give you a single stream (twitter friend feed, etc are all really just one stream) so we thought a differentiator would be the ability to create multiple topic based streams, something power users might want and/or pay for.

    The alternate approach was to try and inject advertising somewhere in our data flow. Twitter, FriendFeed and others will need to find a business model at some point, we've been experimenting with premium services as our model.

    As to twitter loyalty, I'm also curious where the downtime tipping point is where users will leave their system. I think the bar is much higher than most people think.

  • pwb

    Mike's right, Cyndy's wrong. Scaling is *way* easier than creating a super-popular service.

  • Pingback: Twitter and the Importance of Architecture | Mathew Ingram | Voices | AllThingsD

  • Rick

    Please don't feed the enterprise architects… It is very well possible to build a flexible architecture that is cheap and simple to begin with and does scale by gradually replacing parts of it by better designed systems.

    Trying to design for everything up front will put as right back at old skool IT, which never result in a successful online product. Picking the right architecture early sure helps, but there's a difference between being in a world of pain (nobody said it would be easy), and being unable to cope in the way Twitter has.

    In the case of Twitter, looking at were the stand now, there's clearly a management issue, in that it has taken them way too long to start making the necessary changes and getting the right people on board. In most cases, when these changes are made in time, all we experience is a temporary slow-down before the issues are fixed. Let's not forget Twitter is an exception to the rule, lots of successful services are build and scaled the same way without the major outages.

  • mikepk

    Of course it's possible to build simple, scalable architectures but that usually assumes that you know in advance the problem you're trying to solve (or at least the problem space you're in). It's easy to look at Twitter now and editorialize on their scaling issues, hindsight is 20/20. Twitter appears to me to have been started on a lark, a little throwaway side project. They didn't know what they'd built or why anyone would want it, so their initial architecture wasn't trying to solve any particular problem. The interesting thing about startups is that you're sometimes building without knowing what problem you're actually, in the end of the day, solving. Case in point, at Grazr we're still looking for our problem space.

    It's hard to know what's gone on inside Twitter up to this point, but I do agree with you that it's curious that with the time and money they had they didn't fix this problem, or at least see it coming. By the time they got their first round of funding, they already must have seen growth curves, and realized the real problem that that they were solving for their userbase. Then again, until recently, they've been pretty opaque so it's hard to know exactly what went on.

  • Pingback: 2 Birds, 1 Whale: A Story About Twitter

  • Pingback: Everything I Know About Twitter I Learned on Prom Night « tchblg

  • Pingback: That Not So Fresh Feeling » Blog Archive » Hey Twitter! Your problems are my problems!

  • GoFrostfire

    great post, charging was Grazr's big mistake

  • GoFrostfire

    great post, charging was Grazr's big mistake

  • Pingback: Twitter gets cash — can it be fixed? » |

  • Pingback: Should we trust technology? « The Day After Tomorrow

  • Pingback: When It Comes to Social Networks, Uptime Doesn’t Matter

  • Pingback: When It Comes to Social Networks, Uptime Doesn’t Matter | The Click

  • gnoll110 (Noel Kelly)

    Twitter Comment

    @SilkCharm Twitter and the importance of architecture [link to post]

    Posted using Chat Catcher

Older post:

Newer post: