facebooktwittergoogle_plusredditpinterestlinkedinmail

The Twitter guys have been getting a lot of flak over the past few months (and rightly so, in many cases) for the unreliability of their app. But I think they should get some props for opening up and talking about what’s going on over there. Granted, this newfound desire to engage in dialogue (or damage control) should have come a lot earlier, at least in my opinion, but at least they are doing it now. They’ve even managed to foil Mike Arrington’s attempt to start a late-weekend bitchmeme by asking some rather pointed questions of the company.

Although they kind of dance around the specifics a little in their responses, Jack Dorsey and Biz Stone of Twitter seem to more or less confirm many of Mike’s suspicions about their architecture — they admit, for example, that the company is restructuring the way that the system functions, and have brought in some big brains in order to do so:

“We are currently taking a new approach to the way Twitter functions technically with the help of a recently enhanced staff of amazing systems engineers formerly of Google, IBM, and other high-profile technology companies added to our core team.”

I’m not a database guy or a system analyst, but it seems pretty obvious that the structure Twitter has isn’t capable of keeping up with the company’s growth, and may not have been designed properly in the first place to support what has become a gigantic multi-user chat service. In one of the posts on the Twitter dev blog, the company effectively admitted as much. Is that former chief technologist Blaine Cook’s fault? Who knows. But it has to change, and quickly, or Twitter could wind up losing much of the momentum is has gained so far (Michael Krigsman has some good advice on how to proceed).

I was talking with a friend of mine (who is a systems and database expert) about Twitter on the weekend, and he made the point that while startups shouldn’t try to forecast all the ways in which people might use their service — a point that Daniel Burka of Digg made during his recent design presentation at meshU — they still need to pick the right architecture early, or they will be in for a world of pain. In other words, remaining flexible is a good way to approach user-interface design, but not necessarily a good way to approach the design of the actual system structure that makes your service work.

About the author

Mathew 2431 posts

I'm a Toronto-based senior writer with Fortune magazine, and my favorite things to write about are social technology, media and the evolution of online behavior

28 Responses to “Twitter and the importance of architecture”
  1. […] Mathew Ingram takes on what many of us are worrying dealing pissed about, in his new post about determining why Twitter has been so flakey of late. The Twitter guys have been getting a lot of flak over the past few months (and rightly so, in many cases) for the unreliability of their app. But I think they should get some props for opening up and talking about what’s going on over there. Granted, this newfound desire to engage in dialogue (or damage control) should have come a lot earlier, at least in my opinion, but at least they are doing it now. They’ve even managed to foil Mike Arrington’s attempt to start a late-weekend bitchmeme by asking some rather pointed questions of the company. […]

  2. Lucid as always.

  3. Your friend has it exactly right. There's this whole argument that Web 2.0 moves so fast an the result is that companies HAVE to slap something together quickly. The problem is, many of these companies also end up going under just as quickly because they can't handle load. It's all well and good to have something that works great with a few thousand users, but if you want to succeed, you need to think past that. Socialthing is also struggling with this same problem. How many more companies are going to find themselves in this position before they think about getting an enterprise architect to help them design something in the beginning?

  4. Mathew, they screwed up at the get-go. It's original intent was group sharing of SMS messages. The second you think SMS, you have to assume messaging system. SMS IS messaging! I'm not even convinced that if people hadn't started using it the way they do, it would have held under an increased load. At the point at which you add an accessible API, you'd better have a backbone there. So this complaining that they didn't DESIGN it to be used this way is silly. They let it happen. It's like building a house on stilts on an eroding shoreline and then acting confuse when it gets knocked over in a huge storm.

  5. Well I'm going to disagree with you here. As a startup there's only so much energy you have, and you must apportion your resources carefully. The truth is, we like to talk about scaling, but without steady growth and something people find compelling, all the scaling in the world won't help you.

    We significantly over-engineered our architecture at Grazr, spending lots of energy building a powerful and flexible system. We can potentially handle very large streams of data with huge numbers of users, *but* we're not seeing the user adoption that we'd been hoping for. We do many of the things that FriendFeed does, and if we'd been a little looser we could have launched them last October. I even gave a talk at MySQL about this, while it's sexy to work on scaling infrastructure, too much emphasis is a mistake.

    You're seeing now that, even with all their troubles, people are still loyal to Twitter and talking about it. It's definitely painful to change their architecture now, but they have the user momentum to carry them through these tough times. In the balance between building a scalable system, and just getting the users, it's a balance but the latter is more important.

    We've had almost no downtime, have ridden out all of our traffic spikes (from TechCrunch and others) with almost no problems. But no one talks about Grazr with the same passion and loyalty they do Twitter.

    Unless Twitter totally implodes, they'll weather this current pain and the majority of their userbase will stick with them (that's at least my contention). The fact that there are alternatives (and good ones with better stability track records) that have not gained traction should be an interesting indicator of what people care about.

  6. […] best comment on Twitter and architecture I’ve seen It’s the comment left by Michael Kowalchik, aka “MikePK.” He’s the CTO of Grazr and makes an important point that every […]

  7. […] wrote this post to refer back to Michael Kowalchik’s comment on this post. I like his comment too. No matter how well you designed the system architecture, it could still […]

  8. I'm with MikePK (and Scoble). It's easy to criticize Twitter with hindsight, but how did they know how many users and tweets they would have to deal with? More importantly, Twitter, like Grazr, has now written a new chapter in start-up history. Twitter will become legendary for its scaling problems, and for its openness about dealing with these problems.

  9. […] wasn’t expecting this kind of a response from Robert Scoble to my comments on Mathew Ingram’s Blog regarding Twitter and their architectural issues. I really honestly appreciate the feedback on […]

  10. […] Read the rest of this post Print all_things_di220:http://voices.allthingsd.com/20080602/twitter-and-the-importance-of-architecture/ Sphere Comment Tagged: Mike Arrington, Jack Dorsey, Biz Stone, Mathew Ingram, Twitter, Voices | permalink […]

  11. Please don't feed the enterprise architects… It is very well possible to build a flexible architecture that is cheap and simple to begin with and does scale by gradually replacing parts of it by better designed systems.

    Trying to design for everything up front will put as right back at old skool IT, which never result in a successful online product. Picking the right architecture early sure helps, but there's a difference between being in a world of pain (nobody said it would be easy), and being unable to cope in the way Twitter has.

    In the case of Twitter, looking at were the stand now, there's clearly a management issue, in that it has taken them way too long to start making the necessary changes and getting the right people on board. In most cases, when these changes are made in time, all we experience is a temporary slow-down before the issues are fixed. Let's not forget Twitter is an exception to the rule, lots of successful services are build and scaled the same way without the major outages.

  12. […] popular microblogging platform–is down for the count (something that’s been happening a lot lately), they display this charming image on their […]

  13. […] 2, 2008 · No Comments Scoble highlighted a great comment by the CTO of a microblogging service called Grazr.  I’d encourage you to read the full […]

  14. […] All of the problems Twitter is facing seem to be directly related to building the service in a logical fashion, but not forseeing the problems that massive growth would have on the system (a single tweet spawning tens of thousands of database updates). Quite frankly, it’s completely reasonable. Frustrating, but reasonable. As Michael Kowalchik (founder of Grazr) states: As a startup there’s only so much energy you have, and you must apportion your resources carefully. The truth is, we like to talk about scaling, but without steady growth and something people find compelling, all the scaling in the world won’t help you. (mathewingram.com […]

  15. great post, charging was Grazr's big mistake

  16. great post, charging was Grazr's big mistake

  17. […] about how much it goes down. I for one hope that Ev and Biz and the team are busy building a completely separate architecture they can migrate the service over to at some point, because I think Twitter has […]

  18. […] found on micro-blogging application Twitter, which itself is approaching legendary status for its frequent outages. A VC cash infusion to Twitter operators this spring was supposed to correct the technical […]

  19. […] I wrote about Twitter and its downtime last year, I got a fascinating comment from someone who ran a small technology startup — not a Twitter knockoff, but similar in many […]

  20. […] I wrote about Twitter and its downtime last year, I got a fascinating comment from someone who ran a small technology startup — not a Twitter knockoff, but similar in many […]

  21. Twitter Comment


    @SilkCharm Twitter and the importance of architecture [link to post]

    Posted using Chat Catcher

Comments are closed.