The Twitter Engineering Blog

Information from Twitter's engineering team about our technology, tools and events.

Posts from Engineering: scala

The release of Pants 1.0

After five years of development, open source project Pants 1.0.0 is here and we can’t wait for you to check it out.


Finatra 2.0: the fast, testable Scala services framework that powers Twitter

Introducing Finatra 2.0: a high-performance, scalable, testable framework powering production services at Twitter.


Scalding 0.9: Get it while it’s hot!

It’s been just over two years since we open sourced Scalding and today we are very excited to release the 0.9 version. Scalding at Twitter powers everything from internal and external facing dashboards, to custom relevance and ad targeting algorithms, including many graph algorithms such as PageRank, approximate user cosine similarity and many more.

There have been a wide breadth of new features added to Scalding since the last release:


Netty at Twitter with Finagle

Finagle is our fault tolerant, protocol-agnostic RPC framework built atop Netty. Twitter’s core services are built on Finagle, from backends serving user profile information, Tweets, and timelines to front end API endpoints handling HTTP requests.


Observability at Twitter

As Twitter has moved from a monolithic to a distributed architecture, our scalability has increased dramatically.


Streaming MapReduce with Summingbird

Today we are open sourcing Summingbird on GitHub under the ALv2.


Zippy Traces with Zipkin in your Browser

Last summer, we open-sourced Zipkin, a distributed tracing system that helps us gather timing and dependency data for the many services involved in managing requests to Twitter. As we continually improve Zipkin, today we’re adding a Firefox extension to Zipkin that makes it easy to see trace visualizations in your browser as you navigate your website.


Scalding 0.8.0 and Algebird

Earlier this year we open sourced Scalding, a Scala API for Cascading that makes it easy to write big data jobs in a syntax that’s simple and concise. We use Scalding heavily — for everything from custom ad targeting algorithms to PageRank on the Twitter graph.


Building and profiling high performance systems with Iago

Iago is a load generator that we created to help us test services before they encounter production traffic. While there are many load generators available in the open source and commercial software worlds, Iago provides us with capabilities that are uniquely suited for Twitter’s environment and the precise degree to which we need to test our services. There are three main properties that make Iago a good fit for Twitter:


Distributed Systems Tracing with Zipkin

Zipkin is a distributed tracing system that we created to help us gather timing data for all the disparate services involved in managing a request to the Twitter API. As an analogy, think of it as a performance profiler, like Firebug, but tailored for a website backend instead of a browser. In short, it makes Twitter faster.