Twitter and XMPP: Drinking from The Fire Hose

Last year, the folks over at Technorati approached us about building something unique to suit their needs regarding search. We created a special real-time feed of every single public Twitter update to be delivered over the open XMPP protocol. Despite delivery over a faster and cheaper technology, this entire public feed of Twitter updates is resource intensive—we had to be very careful about giving it out.

Technorati no longer uses the feed. However, a few other companies have since approached Twitter and made arrangements with us to gain access. There are four services currently accessing our public timeline feed over XMPP: Twittervision, Zappos, FriendFeed, and Summize.

Dave Troy uses the feed to create Twittervision, an intriguing visual display of Twitter updates based on location. Zappos is parsing the feed for updates from its employees and FriendFeed only requires updates from people who use their service—we can be more efficient in serving these companies’ needs. Summize has greatly improved on Twitter’s Track feature by providing a valuable filtering service using our XMPP feed.

Our goal is to support as many applications, projects, mash-ups, and devices as possible so we'll continue to think about how best to do this. While the XMPP feed of the full Twitter Public Timeline is an amazing resource, drinking from the fire hose is not the best way to quench a thirst. With continued updates and refinement, our API will support most scenarios in a way that preserves overall system performance.

The Twitter API is a big part of our success and we will be stabilizing and improving it in the short term. The rate limit is currently set to 20 requests per hour but we’re working to bring it back to 70. Experimentation with XMPP is informing the future direction of our developer platform. Ultimately, our API will be updated and extended to support a wider variety of interesting projects by a great many talented developers.