The Twitter guys have been getting a lot of flak over the past few months (and rightly so, in many cases) for the unreliability of their app. But I think they should get some props for opening up and talking about what’s going on over there. Granted, this newfound desire to engage in dialogue (or damage control) should have come a lot earlier, at least in my opinion, but at least they are doing it now. They’ve even managed to foil Mike Arrington’s attempt to start a late-weekend bitchmeme by asking some rather pointed questions of the company.

Although they kind of dance around the specifics a little in their responses, Jack Dorsey and Biz Stone of Twitter seem to more or less confirm many of Mike’s suspicions about their architecture — they admit, for example, that the company is restructuring the way that the system functions, and have brought in some big brains in order to do so:

“We are currently taking a new approach to the way Twitter functions technically with the help of a recently enhanced staff of amazing systems engineers formerly of Google, IBM, and other high-profile technology companies added to our core team.”

I’m not a database guy or a system analyst, but it seems pretty obvious that the structure Twitter has isn’t capable of keeping up with the company’s growth, and may not have been designed properly in the first place to support what has become a gigantic multi-user chat service. In one of the posts on the Twitter dev blog, the company effectively admitted as much. Is that former chief technologist Blaine Cook’s fault? Who knows. But it has to change, and quickly, or Twitter could wind up losing much of the momentum is has gained so far (Michael Krigsman has some good advice on how to proceed).

I was talking with a friend of mine (who is a systems and database expert) about Twitter on the weekend, and he made the point that while startups shouldn’t try to forecast all the ways in which people might use their service — a point that Daniel Burka of Digg made during his recent design presentation at meshU — they still need to pick the right architecture early, or they will be in for a world of pain. In other words, remaining flexible is a good way to approach user-interface design, but not necessarily a good way to approach the design of the actual system structure that makes your service work.

About the author

Mathew 2406 posts

I'm a Toronto-based former senior writer with Gigaom and my favorite things to write about are social technology, media and the evolution of online behavior

28 Responses to “Twitter and the importance of architecture”
  1. […] Mathew Ingram takes on what many of us are worrying dealing pissed about, in his new post about determining why Twitter has been so flakey of late. The Twitter guys have been getting a lot of flak over the past few months (and rightly so, in many cases) for the unreliability of their app. But I think they should get some props for opening up and talking about what’s going on over there. Granted, this newfound desire to engage in dialogue (or damage control) should have come a lot earlier, at least in my opinion, but at least they are doing it now. They’ve even managed to foil Mike Arrington’s attempt to start a late-weekend bitchmeme by asking some rather pointed questions of the company. […]

  2. Lucid as always.

  3. Your friend has it exactly right. There's this whole argument that Web 2.0 moves so fast an the result is that companies HAVE to slap something together quickly. The problem is, many of these companies also end up going under just as quickly because they can't handle load. It's all well and good to have something that works great with a few thousand users, but if you want to succeed, you need to think past that. Socialthing is also struggling with this same problem. How many more companies are going to find themselves in this position before they think about getting an enterprise architect to help them design something in the beginning?

    • i think you're right, Cyndy — and you had a good post recently at
      Profy along the same lines. But I guess the tough part for companies
      like Twitter is, what happens if the way people use your app doesn't
      jibe with the way that you designed the architecture? In other words,
      whose fault is it if you design a system to do content management and
      people wind up using it as a global messaging system? Maybe it's no
      one's fault. As usual, I blame Scoble :-)

  4. Mathew, they screwed up at the get-go. It's original intent was group sharing of SMS messages. The second you think SMS, you have to assume messaging system. SMS IS messaging! I'm not even convinced that if people hadn't started using it the way they do, it would have held under an increased load. At the point at which you add an accessible API, you'd better have a backbone there. So this complaining that they didn't DESIGN it to be used this way is silly. They let it happen. It's like building a house on stilts on an eroding shoreline and then acting confuse when it gets knocked over in a huge storm.

  5. Well I'm going to disagree with you here. As a startup there's only so much energy you have, and you must apportion your resources carefully. The truth is, we like to talk about scaling, but without steady growth and something people find compelling, all the scaling in the world won't help you.

    We significantly over-engineered our architecture at Grazr, spending lots of energy building a powerful and flexible system. We can potentially handle very large streams of data with huge numbers of users, *but* we're not seeing the user adoption that we'd been hoping for. We do many of the things that FriendFeed does, and if we'd been a little looser we could have launched them last October. I even gave a talk at MySQL about this, while it's sexy to work on scaling infrastructure, too much emphasis is a mistake.

    You're seeing now that, even with all their troubles, people are still loyal to Twitter and talking about it. It's definitely painful to change their architecture now, but they have the user momentum to carry them through these tough times. In the balance between building a scalable system, and just getting the users, it's a balance but the latter is more important.

    We've had almost no downtime, have ridden out all of our traffic spikes (from TechCrunch and others) with almost no problems. But no one talks about Grazr with the same passion and loyalty they do Twitter.

    Unless Twitter totally implodes, they'll weather this current pain and the majority of their userbase will stick with them (that's at least my contention). The fact that there are alternatives (and good ones with better stability track records) that have not gained traction should be an interesting indicator of what people care about.

    • That's an excellent point, Mike. If a startup had to choose between
      your problems (not enough users) and Twitter's (passionate users but
      the wrong architecture), I assume that most would probably choose the
      latter. I guess the hard part is that not many companies get to choose
      – they have to anticipate, which is always going to be inaccurate in
      a variety of ways. But your point is a good one: don't spend too much
      time on architecture, or you could wind up with a beautifully built
      and sturdy structure, but no users.

      • Mike's right, Cyndy's wrong. Scaling is *way* easier than creating a super-popular service.

    • Grazr broke the cardinal rule of Web 2.0: charging a fee. The free account is limited with the one stream and the paid accounts are $10 a month: steep in this economy for facilitating widespread adoption. It has nothing to do with who got out of the gate first; it has to do with which one was free. It's like comparing free apples with $1 a piece oranges. Everyone is going to take an apple even if they don't like them better.

      As for loyal users, how loyal are they? It would be interesting to see if my experience is the same as others; I'm spending far more time leaving comments on blog posts about Twitter than I am actually using the service. I'm very reliant on the IM portion of Twitter to actually use it, an even Twhirl, my second choice, horks every 2 seconds as “over limit.” Which means my use is steadily declining. If some one came along with a true alternative that made it easy to port data, I'm wondering how far that loyalty would go.

      • I see your point, but we already have a lot of free services that we've been offering. Hosting reading lists, the ability to create free feed widgets, and several others, that when it came to releasing streams we decided to try and charge for something. All the other services only give you a single stream (twitter friend feed, etc are all really just one stream) so we thought a differentiator would be the ability to create multiple topic based streams, something power users might want and/or pay for.

        The alternate approach was to try and inject advertising somewhere in our data flow. Twitter, FriendFeed and others will need to find a business model at some point, we've been experimenting with premium services as our model.

        As to twitter loyalty, I'm also curious where the downtime tipping point is where users will leave their system. I think the bar is much higher than most people think.

  6. […] best comment on Twitter and architecture I’ve seen It’s the comment left by Michael Kowalchik, aka “MikePK.” He’s the CTO of Grazr and makes an important point that every […]

  7. […] wrote this post to refer back to Michael Kowalchik’s comment on this post. I like his comment too. No matter how well you designed the system architecture, it could still […]

  8. I'm with MikePK (and Scoble). It's easy to criticize Twitter with hindsight, but how did they know how many users and tweets they would have to deal with? More importantly, Twitter, like Grazr, has now written a new chapter in start-up history. Twitter will become legendary for its scaling problems, and for its openness about dealing with these problems.

  9. […] wasn’t expecting this kind of a response from Robert Scoble to my comments on Mathew Ingram’s Blog regarding Twitter and their architectural issues. I really honestly appreciate the feedback on […]

  10. […] Read the rest of this post Print all_things_di220:http://voices.allthingsd.com/20080602/twitter-and-the-importance-of-architecture/ Sphere Comment Tagged: Mike Arrington, Jack Dorsey, Biz Stone, Mathew Ingram, Twitter, Voices | permalink […]

  11. Please don't feed the enterprise architects… It is very well possible to build a flexible architecture that is cheap and simple to begin with and does scale by gradually replacing parts of it by better designed systems.

    Trying to design for everything up front will put as right back at old skool IT, which never result in a successful online product. Picking the right architecture early sure helps, but there's a difference between being in a world of pain (nobody said it would be easy), and being unable to cope in the way Twitter has.

    In the case of Twitter, looking at were the stand now, there's clearly a management issue, in that it has taken them way too long to start making the necessary changes and getting the right people on board. In most cases, when these changes are made in time, all we experience is a temporary slow-down before the issues are fixed. Let's not forget Twitter is an exception to the rule, lots of successful services are build and scaled the same way without the major outages.

    • Of course it's possible to build simple, scalable architectures but that usually assumes that you know in advance the problem you're trying to solve (or at least the problem space you're in). It's easy to look at Twitter now and editorialize on their scaling issues, hindsight is 20/20. Twitter appears to me to have been started on a lark, a little throwaway side project. They didn't know what they'd built or why anyone would want it, so their initial architecture wasn't trying to solve any particular problem. The interesting thing about startups is that you're sometimes building without knowing what problem you're actually, in the end of the day, solving. Case in point, at Grazr we're still looking for our problem space.

      It's hard to know what's gone on inside Twitter up to this point, but I do agree with you that it's curious that with the time and money they had they didn't fix this problem, or at least see it coming. By the time they got their first round of funding, they already must have seen growth curves, and realized the real problem that that they were solving for their userbase. Then again, until recently, they've been pretty opaque so it's hard to know exactly what went on.

  12. […] popular microblogging platform–is down for the count (something that’s been happening a lot lately), they display this charming image on their […]

  13. […] 2, 2008 · No Comments Scoble highlighted a great comment by the CTO of a microblogging service called Grazr.  I’d encourage you to read the full […]

  14. […] All of the problems Twitter is facing seem to be directly related to building the service in a logical fashion, but not forseeing the problems that massive growth would have on the system (a single tweet spawning tens of thousands of database updates). Quite frankly, it’s completely reasonable. Frustrating, but reasonable. As Michael Kowalchik (founder of Grazr) states: As a startup there’s only so much energy you have, and you must apportion your resources carefully. The truth is, we like to talk about scaling, but without steady growth and something people find compelling, all the scaling in the world won’t help you. (mathewingram.com […]

  15. great post, charging was Grazr's big mistake

  16. great post, charging was Grazr's big mistake

  17. […] about how much it goes down. I for one hope that Ev and Biz and the team are busy building a completely separate architecture they can migrate the service over to at some point, because I think Twitter has […]

  18. […] found on micro-blogging application Twitter, which itself is approaching legendary status for its frequent outages. A VC cash infusion to Twitter operators this spring was supposed to correct the technical […]

  19. […] I wrote about Twitter and its downtime last year, I got a fascinating comment from someone who ran a small technology startup — not a Twitter knockoff, but similar in many […]

  20. […] I wrote about Twitter and its downtime last year, I got a fascinating comment from someone who ran a small technology startup — not a Twitter knockoff, but similar in many […]

  21. Twitter Comment


    @SilkCharm Twitter and the importance of architecture [link to post]

    Posted using Chat Catcher

Comments are closed.