Forward Secrecy at Twitter

As part of our continuing effort to keep our users’ information as secure as possible, we’re happy to announce that we recently enabled forward secrecy for traffic on twitter.com, api.twitter.com, and mobile.twitter.com. On top of the usual confidentiality and integrity properties of HTTPS, forward secrecy adds a new property. If an adversary is currently recording all Twitter users’ encrypted traffic, and they later crack or steal Twitter’s private keys, they should not be able to use those keys to decrypt the recorded traffic. As the Electronic Frontier Foundation points out, this type of protection is increasingly important on today’s Internet.

Under traditional HTTPS, the client chooses a random session key, encrypts it using the server’s public key, and sends it over the network. Someone in possession of the server’s private key and some recorded traffic can decrypt the session key and use that to decrypt the entire session. In order to support forward secrecy, we’ve enabled the EC Diffie-Hellman cipher suites. Under those cipher suites, the client and server manage to come up with a shared, random session key without ever sending the key across the network, even under encryption. The details of this remarkable and counterintuitive key exchange are explained at Wikipedia’s excellent article on Diffie-Hellman key exchange. The server’s private key is only used to sign the key exchange, preventing man-in-the-middle attacks.

There are two main categories of Diffie-Hellman key exchange. Traditional Diffie-Hellman (DHE) depends on the hardness of the Discrete Logarithm Problem and uses significantly more CPU than RSA, the most common key exchange used in SSL. Elliptic Curve Diffie-Hellman (ECDHE) is only a little more expensive than RSA for an equivalent security level. Vincent Bernat (@vince2_benchmarked ECDHE at a 15% overhead relative to RSA over 2048-bit keys. DHE, by comparison, used 310% more CPU than RSA.

Forward secrecy is just the latest way in which Twitter is trying to defend and protect the user’s voice.

In practical deployment, we found that enabling and prioritizing ECDHE cipher suites actually caused negligible increase in CPU usage. HTTP keepalives and session resumption mean that most requests do not require a full handshake, so handshake operations do not dominate our CPU usage. We find 75% of Twitter’s client requests are sent over connections established using ECDHE. The remaining 25% consists mostly of older clients that don’t yet support the ECDHE cipher suites.

The last obstacle to correctly implementing forward secrecy was our use of TLS session tickets. We use TLS session tickets to allow clients to reconnect quickly using an abbreviated handshake if they still have a session ticket from a recent connection. Beside the CPU savings, this saves one network round-trip, commonly around 150ms and often much more on mobile networks.

Session tickets enable pools of servers to support session resumption without need for a shared session cache. However, as Adam Langley (@agl__) points out in his blog post How to Botch TLS Forward Secrecy:

If you run several servers then they all need to share the same session ticket key otherwise they can’t decrypt each other’s session tickets. In this case your forward secrecy is limited by the security of that file on disk. Since your private key is probably kept on the same disk, enabling forward secure cipher suites probably hasn’t actually achieved anything other than changing the file that the attacker needs to steal!

You need to generate session ticket keys randomly, distribute them to the servers without ever touching persistent storage and rotate them frequently.

We implemented such a key rotation system with a few additional goals: It should be simple to implement, simple to maintain, resistant to failure, and we should be able to restart the frontend process during deploys without waiting to receive new session ticket keys.

To do so, we have a set of key generator machines, of which one is the leader. The leader generates a fresh session ticket key every twelve hours and zeroes old keys after thirty-six hours. Keys are stored in tmpfs (a RAM-based filesystem), with no swap partitions configured. It’s important that there be no swap, because tmpfs will use swap if available, which could write keys to long-term disk storage.

Every five minutes, our frontends fetch the latest ticket keys from a key generator machine via SSH. This traffic is also forward secret by SSH’s Diffie-Hellman key exchange. Again, the keys are stored on a tmpfs, so they survive a restart of the frontend process without being leaked to disk.

When a new ticket key K is available, we don’t want to start encrypting new tickets with it until we expect that each frontend has a copy of K and can decrypt those tickets. We use a timestamp in K’s filename to determine its age; once it is at least twenty minutes old, it is considered current and frontends will start encrypting tickets with it.

In general, frontends will have three ticket keys available: the current key, and the two previous keys. They can decrypt session tickets using any of those three. When a client resumes a session using one of the previous keys, a frontend will finish the abbreviated handshake and assign a new session ticket using the current key. This logic is implemented in a callback provided to OpenSSL’s SSL_CTX_set_tlsext_ticket_key_cb.

The New Normal

At the end of the day, we are writing this not just to discuss an interesting piece of technology, but to present what we believe should be the new normal for web service owners. A year and a half ago, Twitter was first served completely over HTTPS. Since then, it has become clearer and clearer how important that step was to protecting our users’ privacy.

If you are a webmaster, we encourage you to implement HTTPS for your site and make it the default. If you already offer HTTPS, ensure your implementation is hardened with HTTP Strict Transport Security, secure cookies, certificate pinning, and Forward Secrecy. The security gains have never been more important to implement.

If you don’t run a website, demand that the sites you use implement HTTPS to help protect your privacy, and make sure you are using an up-to-date web browser so you are getting the latest security improvements.

Security is an ever-changing world. Our work on deploying forward secrecy is just the latest way in which Twitter is trying to defend and protect the user’s voice in that world.

Acknowledgements

Like all projects at Twitter, this has been a collaboration. We’d like to thank all involved, including: Jeff Hodges, Jeffrey Pinner, Kyle Randolph, Jan Schaumann, Matt Finifter, the entirety of our Security and Twitter Frontend teams, and the broader security and cryptography communities.