For the second time, we had the opportunity to participate in the Google Summer of Code (GSoC) and we want to share news on the resulting open source activities. Unlike many GSoC participating organizations that focus on a single ecosystem, we have a variety of projects that span multiple programming languages and communities.
We worked on three projects with three amazing students over the summer.
Tomas Tauber (@shoguncz) worked closely with his mentor Oscar Boykin (@posco) to improve the performance of Scalding by adding matrix optimizations (see the commits). For example, how should we multiply A*B*C? Perhaps (A*B)*C takes a lot longer than A*(B*C) due to the sizes of the matrices. What about matrices with huge skew, such as Twitter’s follower graph where some users have millions of followers, but most have only a handful? By optimizing at the Matrix API layer, we can easily reap the benefits at higher layers. This project added a scheduler to the formulas users write with Matrices and performs the computation in the optimal order, where optimal is in terms of intermediate data size and formula tree-depth. See the performance results for more information.
Mohamed Bakkar worked with the lead of the Netty project Trustin Lee (@trustin) to add a built-in asynchronous DNS resolver. Instead of using the blocking DNS resolver provided by JDK, the new resolver will prevent applications built on top of Netty from their performance being impacted by slow or overloaded DNS servers.
Ilim Ugur (@ilimugur) worked with Mesos committer Vinod Kone (@vinodkone) to add an authentication stage in Mesos before letting frameworks and slaves talk to the master(s) thereby making the communication between the modules forming Mesos (masters, slaves and frameworks) more secure.
As part of GSoC, students and mentoring organizations receive a stipend. We are donating our portion of the stipend to the Software Freedom Conservancy (@conservancy) which is a 501(c)(3) organization that helps provide a non-profit home and infrastructure for open source projects like Git and Selenium.
We really enjoyed the opportunity to take part in Google Summer of Code. Thanks again to our three students, mentors and to Google for the program. We look forward to supporting it next year.