Announcing gzip compression for Streaming APIs

A piece of feedback we regularly get from Streaming API consumers is that the bandwidth required to maintain a connection can be very large for high-traffic connections such as the Firehose and Site Streams. As the number of Tweets flowing through Twitter’s servers grows, this has been a bigger and bigger problem. I’m happy to announce that we’ve started rolling out code which will help address this, by sending gzipped streams to consumers who request them.

When rolled out, you will be able to request a gzipped stream by connecting to a stream with the following HTTP header:

Accept-Encoding: deflate, gzip

This will indicate to us that we can deliver a gzipped stream, which, from some testing, has been up to 1/5th the size of an uncompressed stream- a considerable savings of bandwidth!

Our rollout plan is to first deploy this to Site Streams next week. If that goes well, we’ll update stream.twitter.com with this capability soon after. If you are not currently sending the appropriate Accept-Encoding header, then you should be unaffected by this change.