While we’re at the Percona Live MySQL Conference, we’d like to discuss updates on how Twitter uses MySQL, as well as share our plans to open source Mysos, a new MySQL on Apache Mesos framework.
MySQL at Twitter
Since Twitter was founded, MySQL has been one of our key data storage technologies. We store data in hundreds of schemas and our largest cluster is thousands of nodes serving millions of queries per second. At the scale of Twitter, we are pushing MySQL to its limits.
At Twitter, MySQL is used in two ways:
As Twitter has grown as a service, so has our need to scale storage technologies like MySQL, PostgreSQL, Vertica and Manhattan in production. To make this possible, we’re actively hiring for positions including database administrators, site reliability engineers and software engineers working on distributed storage technology.
Contributing to the open source community
Twitter has benefited greatly from the MySQL community and we’ve contributed many patches back upstream. Examples of patches contributed to MySQL include:
The goal of WebScaleSQL is to enable the scale-oriented members of the MySQL community to work closer to build and add more features to MySQL that are specific to deployments in large scale environments. You can check out some of our examples of patches contributed to WebScaleSQL here.
Introducing and incubating Mysos
In an effort to improve the scalability and management of our MySQL clusters, we’ve begun work on a new framework called Mysos. The Mysos project leverages Apache Mesos to build a scalable database service for MySQL. Mesos provides primitives to allow Mysos to reliably schedule, monitor and communicate with MySQL instances. As a storage framework, Mysos will be able to use recently-added persistent storage primitives within Mesos. It dramatically simplifies the management of a MySQL cluster and is designed to offer:
Having initially developed Mysos, we’re now announcing our plans to open source the project and our intention to build a strong, independent open source community around Mysos. We are still in the early stages and the code isn’t meant for production usage yet. However, we are starting to seed the code and a proposal at the Apache Incubator with engineers from Twitter, Mesosphere and the Apache Mesos community. We invite anyone interested in scaling MySQL on Mesos to reach out to the Mysos community — we’d love to have you involved. At the moment, we’re best reached by visiting the #mysos IRC channel on Freenode.
The Mysos project was born out of a collaboration between members of the Cloud Infrastructure team at Twitter (Yan Xu, Chris Lambert, Vinod Kone, Dominic Hamon, Jie Yu, Ben Mahler, Brian Wickman), members of the @TwitterOSS team (Chris Aniszczyk, Dave Lester), members of the @TwitterDBA team (Chuck Sumner, Jonah Berquist, Pascal Borghino), and the MySQL team (Calvin Sun, Inaam Rana, Tugrul Bingol).