Engineering Back

Discover: Improved personalization algorithms and real-time indexing

By
Tuesday, 1 May 2012

We are beginning to roll out a new version of the Discover tab that is even more personalized for you. We’ve improved our personalization algorithms to incorporate several new signals including the accounts you follow and whom they follow. All of this social data is used to understand your interests and display stories that are relevant to you in real-time.

Behind the scenes, the new Discover tab is powered by Earlybird, Twitter’s real-time search technology. When a user tweets, that Tweet is indexed and becomes searchable in seconds. Every Tweet with a link also goes through some additional processing: we extract and expand any URLs available in Tweets, and then fetch the contents of those URLs via SpiderDuck, our real-time URL fetcher.

To generate the stories that are based on your social graph and that we believe are most interesting to you, we first use Cassovary, our graph processing library, to identify your connections and rank them according to how strong and important those connections are to you.

Once we have that network, we use Twitter’s flexible search engine to find URLs that have been shared by that circle of people. Those links are converted into stories that we’ll display, alongside other stories, in the Discover tab. Before displaying them, a final ranking pass re-ranks stories according to how many people have tweeted about them and how important those people are in relation to you. All of this happens in near-real time, which means breaking and relevant stories appear in the new Discover tab almost as soon as people start talking about them.

Our NYC engineering team, led by Daniel Loreto (@DanielLoreto), along with Julian Marinus (@fooljulian), Alec Thomas (@alecthomas), Dave Landau (@landau), and Ugo Di Girolamo (@ugodiggi), is working hard on Discover to create new ways to bring you instantly closer to the things you care about. This update is just the beginning of this ongoing effort.

- Ori Allon, Director of Engineering (@oriallon)