The Twitter Engineering Blog

Information from Twitter's engineering team about our technology, tools and events.

Results from Engineering for: 2014

Building a complete Tweet index

Today, we are pleased to announce that Twitter now indexes every public Tweet since 2006.

Since that first simple Tweet over eight years ago, hundreds of billions of Tweets have captured everyday human experiences and major historical events. Our search engine excelled at surfacing breaking news and events in real time, and our search index infrastructure reflected this strong emphasis on recency. But our long-standing goal has been to let people search through every Tweet ever published.


Breakout detection in the wild

A novel statistically rigorous approach for early detection of breakout(s) in web-scale cloud systems.


Investing in MIT’s new Laboratory for Social Machines

Today, @MIT announced the creation of the Laboratory for Social Machines, funded by a


Celebrating over a year of @FlightJS

FlightJS is an evolving JavaScript framework project. See what’s happened since we initially open-sourced it over a year ago.


Hello Pants build

The story behind a new open source build system written in Python.


Push our limits - reliability testing at Twitter

At Twitter, we strive to prepare for sustained traffic as well as spikes - some of which we can plan for, some of which comes at unexpected times or in unexpected ways. To help us prepare for these varied types of traffic, we continuously run tests against our infrastructure to ensure it remains a scalable and highly available system.


All-pairs similarity via DIMSUM

Given a dataset of sparse vector data, we solve the problem of finding all similar vector pairs according to a similarity function.


Bringing more design principles to security

To date, much of the web and mobile security focus has been on security bugs such as cross-site-scripting and SQL injection. Due to the number of those issues and the fact that the number of bugs in general increases in proportion to the number of lines of code, it’s clear that if we hope to address software security problems as a community, we also need to invest in designing software securely to eliminate entire classes of bugs.


Outreach Program for Women and GSoC 2014 results

We worked on nine open source projects this summer with nine incredible students.


Fighting spam with BotMaker

To fight spam on Twitter, we built BotMaker — a system we designed and implemented from the ground up. BotMaker provides a solid foundation for our principled defense against unsolicited content on Twitter.