The Twitter Engineering Blog

Information from Twitter's engineering team about our technology, tools and events.

Results from Engineering for: 2011

SpiderDuck: Twitter's Real-time URL Fetcher

Tweets often contain URLs or links to a variety of content on the web, including images, videos, news articles and blog posts. SpiderDuck is a service at Twitter that fetches all URLs shared in Tweets in real-time, parses the downloaded content to extract metadata of interest and makes that metadata available for other Twitter services to consume within seconds.

Several teams at Twitter need to access the linked content, typically in real-time, to improve Twitter products. For example:


Twitter’s mobile web app delivers performance

As the number of people using Twitter has grown, we’ve wanted to make sure that we deliver the best possible experience to users, regardless of platform or device. Since is not optimized for smaller screens or touch interactions familiar to many smart phones, we decided to build a cross-platform web application that felt native in its responsiveness and speed for those who prefer accessing Twitter on their phone’s or the tablet’s browser.


Finagle: A Protocol-Agnostic RPC System

Finagle is a protocol-agnostic, asynchronous RPC system for the JVM that makes it easy to build robust clients and servers in Java, Scala, or any JVM-hosted language.


A Storm is coming: more details and plans for release

We’ve received a lot of questions about what’s going to happen to Storm now that BackType has been acquired by Twitter. I’m pleased to announce that I will be releasing Storm at Strange Loop on September 19th! Check out the session info for more details.


Fast Core Animation UI for the Mac

Starting today, Twitter is offering TwUI as an open-source framework ( for developing interfaces on the Mac.


Join the Flock!

Twitter’s engineering team is growing quickly. Two-thirds of our engineers were hired in the last 12 months. Those engineers joined us from cities and countries around the world and from companies of various sizes.


The Engineering Behind Twitter’s New Search Experience

Today, Twitter launched a personalized search experience to help our users find the most relevant Tweets, images, and videos. To build this product, our infrastructure needed to support two major features: relevance-filtering of search results and the identification of relevant images and photos.


Faster Ruby: Kiji Update

In March 2011, we shared Kiji, an improved Ruby runtime. The initial performance gains were relatively modest, but laid the foundation for future improvements. We continued the work and now have some excellent results.


Twitter Search is Now 3x Faster

In the spring of 2010, the search team at Twitter started to rewrite our search engine in order to serve our ever-growing traffic, improve the end-user latency and availability of our service, and enable rapid development of new search features. As part of the effort, we launched a new real-time search engine, changing our back-end from MySQL to a real-time version of Lucene.


Improving Browser Security with CSP

If you are using Firefox 4, you now have an extra layer of security when accessing