Tuesday, April 20, 2010 | By wan li yang (@wanliyang) [16:40 UTC]
At Twitter, we use memcached to speed up page loads and alleviate database load. We have many memcached hosts. To make our system robust, our memcached clients use consistent hashing and enable the auto_eject_hosts option. With this many hosts and this kind of configuration, one would assume that it won’t be noticeable if one memcached host goes down, right?
Thursday, April 8, 2010 | By Kevin Weil (@kevinweil) [21:10 UTC]
My name is Kevin Weil and I’m a member of the analytics team at Twitter. We’re collectively responsible for Twitter’s data warehousing, for building out an analysis platform that lets us easily and efficiently run large calculations over the Twitter dataset, and ultimately for turning that data into something actionable that helps the business. We’re fortunate to work with great people from teams across the organization for the latter.
Many modern web sites need fast access to an amount of information so large that it cannot be efficiently stored on a single computer. A good way to deal with this problem is to “shard” that information; that is, store it across multiple computers instead of on just one.