All about Apache Aurora

Thursday, 12 February 2015

Today we’re excited to see the Apache Aurora community announce the 0.7.0 release. Since we began development on this project, Aurora has become a critical part of how we run services at Twitter. Now a fledgling open source project, Aurora is actively developed by a community of developers running it in production.

This is Aurora’s third release since joining the Apache Incubator. New features include beta integration with Docker – which allows developers to deploy their pre-packaged applications as lightweight containers – as well as full support for a new command-line client that makes it even simpler to deploy services via Aurora. A full changelog is available documenting more updates.

What is Aurora?

Platforms like Twitter operate across tens of thousands of machines, with hundreds of engineers deploying software daily. In this type of environment, automation is critical. Aurora is software that keeps services running in the face of many types of failure, and provides engineers a convenient, automated way to create and update these services.

To accomplish this, Aurora leverages the Apache Mesos cluster manager, which provides information about the state of the cluster. Aurora uses that knowledge to make scheduling decisions. For example, when a machine experiences failure Aurora automatically reschedules those previously-running services onto a healthy machine in order to keep them running.

History of Aurora at Twitter

Development on Aurora began in 2010 when Bill Farner, an engineer on Twitter’s research team, started a project to simplify operations of the many services making up Twitter’s architecture. Bill formerly worked at Google and wanted the project to achieve similar goals as Borg, Google’s cluster manager.

Over time, a growing team of Twitter engineers began to contribute directly to and rely on Aurora. Additionally, Aurora began to host key Twitter services such as Twitter’s ad-serving platform. Today, all new stateless services – and most existing services – run on Aurora.

Not only does Aurora assist with keeping the platform running, Aurora has helped push utilization of our cluster. This has allowed us to run more applications on fewer servers and substantially save costs while increasing efficiency.

A growing open source community

In late 2013, we shared our work with the open source community by releasing Aurora into the Apache Incubator.

Since then, Aurora has been cultivating a vibrant and growing community. The @ApacheAurora community holds weekly IRC meetings open to the public for participation. To get involved, you can join #aurora on irc.freenode.net every Monday at 11 a.m. PT.

If you’d like to learn more about Aurora and how it’s used at huge scale in production, we’re hosting an Aurora meetup at Twitter HQ on Thursday, Feb. 19.

This meetup will feature talks from Steve Niemitz (@steveniemitz) from TellApart, who contributed to the Docker integration in the latest release, well as Joe Smith, one of the small number of SREs at Twitter who run Aurora. We’d love to see you there.

Besides Twitter, many companies are currently using Aurora.

Acknowledgements

Many people have worked to make Aurora possible both across Twitter and the greater community. Special thanks to the release manager for the 0.7.0 release, Maxim Khutornenko, and the many other contributors for this release.