Today, we are pleased to announce that Twitter now indexes every public Tweet since 2006.
Since that first simple Tweet over eight years ago, hundreds of billions of Tweets have captured everyday human experiences and major historical events. Our search engine excelled at surfacing breaking news and events in real time, and our search index infrastructure reflected this strong emphasis on recency. But our long-standing goal has been to let people search through every Tweet ever published.
At Twitter, we strive to prepare for sustained traffic as well as spikes - some of which we can plan for, some of which comes at unexpected times or in unexpected ways. To help us prepare for these varied types of traffic, we continuously run tests against our infrastructure to ensure it remains a scalable and highly available system.
To date, much of the web and mobile security focus has been on security bugs such as cross-site-scripting and SQL injection. Due to the number of those issues and the fact that the number of bugs in general increases in proportion to the number of lines of code, it’s clear that if we hope to address software security problems as a community, we also need to invest in designing software securely to eliminate entire classes of bugs.
To fight spam on Twitter, we built BotMaker — a system we designed and implemented from the ground up. BotMaker provides a solid foundation for our principled defense against unsolicited content on Twitter.