Connecting to the pulse of the planet with Twitter APIs

Tuesday, 27 May 2014

Twitter is a global, real-time platform; an indispensable companion to life in the moment. As such, Twitter is a reflection of what’s happening in the world at any given time and the Twitter APIs give developers the opportunity to tap into the pulse of the planet and use that data in innovative ways.

I recently had the pleasure of giving a talk at the API Strategy & Practice Conference in Amsterdam to a fantastic group of developers. After sharing a few stories about Twitter and highlighting some companies in the Twitter ecosystem, I presented the APIs and performed live demos to show how easy it is to get started using the Twitter platform.

Twitter Streaming APIs

The Twitter platform consists of two major set of APIs: the Streaming APIs, which allow you to access to Tweets in real time, and the REST APIs, which make it possible for you to perform actions on Twitter and tap into the existing set of data.

Streaming APIs are my personal favorite because they are the very essence of “the pulse of the planet.” They materialize as a long-lived HTTP connection with a very low latency, sending new Tweets as they are posted in a fraction of a second so you can see what’s happening at this precise moment. In fact, Streaming APIs are at the cornerstone of the most fascinating products relying on Twitter data.

Analyzing the mood of people in the world, predicting elections, detecting breaking news or providing help to people during emergencies: these are just a few examples of the remarkable possibilities offered by the Twitter platform.

Real-time demos with Node.js and WebSockets

The main goal of the demos in this keynote was to show both the power and the simplicity of consuming data from the Streaming APIs. They are entirely written in JavaScript, using Node.js to connect to Twitter APIs and WebSockets to communicate with the browser in order to reflect the real-time nature of Twitter.

Monitoring and analyzing Tweets

There are a few different ways to connect to the Streaming APIs. A good place to start is the sample endpoint, which provides you with roughly 1% of all Tweets, randomly selected from the firehose. There are 500 million Tweets posted every single day, so 1% is still a very large number and statistically relevant for many use cases.

This first demo is extremely simple. It gathers Tweets from the sample endpoint, and every time a Tweet is received, the information gets sent from Node to the web page over a WebSocket in order to update a count.

Connecting to the pulse of the planet with Twitter APIs

This gives a glimpse into the amount of Tweets received, representing a sample to perform data analysis projects. As you can see on the screenshot, you can expect on average between 50 and 60 Tweets per second (TPS) when connecting to the sample endpoint. We see on average 5,700 TPS on the platform, with a record at 143,199 TPS, which is why the complete firehose is very hard to consume without a high-capacity platform behind an application.

Adding context and filtering Tweets

Beyond the sample, one of the most interesting way to collect Tweets in real time is the filter endpoint, which lets you receive Tweets matching different criteria. Whether you want to track Tweets with some keywords, Tweets from a selection of users, or Tweets posted around a location when users explicitly choose to share it, this endpoint is fascinating since it makes it possible to connect to the pulse of the planet in a way that is relevant to a very specific context.

The following demo provides a simple interface for the different parameters offered by this endpoint. By sending the selected options such as a hashtag and a bounding box on the map over a WebSocket, the Node server establishes a connection to the filter endpoint which starts streaming Tweets matching at least one of the criteria. For example, on the screenshot below, Tweets containing #WorldCup or sent from a region in Brazil are sent over the WebSocket and displayed on the web page.

Connecting to the pulse of the planet with Twitter APIs

Going beyond the browser

With the rise of the mobile ecosystem, the Twitter APIs have become more important than ever when building mobile apps. In fact, taking the personal context of the phone, such as the user’s location, weather conditions or calendar, and combining it with the user’s social context through Twitter APIs enables you to build great mobile experiences that will delight your users.

As Twitter has evolved into a universal platform to share, consume and broadcast information, it is also perfectly suited for connected devices and beyond. For instance, we’ve seen hundreds of radio-tagged sharks tweet warnings for swimmers safety in Australia.

Tweeting a picture from a Raspberry Pi

All API Strategy conference attendees received a Raspberry Pi. In order for them to make the most of this gift and demonstrate how ubiquitous Twitter is, I bought the tiny Camera Module and transformed my Raspberry Pi into a small connected device that can tweet pictures.

The demo consists in interacting with both the camera and the Twitter APIs using Node running on the device, sharing the connection from a computer so it can access the Internet. Below you can see one of the pictures taken from stage by the Raspberry Pi and immediately uploaded to the Twitter using the corresponding API endpoint:

Controlling a drone from Tweets

Last but not least, the bonus demo for this talk involved an AR.Drone quadcopter. By connecting my Mac both to the drone via its own WiFi network and to the Internet via Bluetooth, tethering 3G from my phone, the great folks in the audience were able to control the drone from Tweets!

This is actually fairly simple to achieve by using the amazing work from Felix and the NodeCopter community to control a drone from JavaScript. Connecting to the filter streaming endpoint described earlier to listen to @twicopter mentions or the #apistrat hashtag, you can make the quadcopter take off from Tweets, and also reply to the Tweets sent by users to acknowledge their commands.

Code, video and slides

You can find code for the demos on GitHub. You can access the video of my talk and the slide deck for more details. You can read the story of the talk (and reactions from the audience) on this custom timeline.

We hope this talk and demos will inspire you and help you come up with creative ideas on how to leverage Twitter APIs in new innovative ways.

Thanks and upcoming events

Thank you very much to Steven Willmott, Kin Lane, the 3scale team, speakers and everyone who attended, it was great to connect with you all in Amsterdam! The next edition of the API Strategy Conference will take place in Chicago on September 24, 25 and 26.

I will be speaking at APIdays Barcelona on May 29-30 and push the boundaries of those Twitter platform demos. Don’t forget to check out our Twitter page on Lanyrd to follow the Developer Relations team, we would love to meet you at an upcoming event!

To get started with the Twitter platform, check out our documentation. If you have any questions, please don’t hesitate to tweet to Romain.