The Twitter Engineering Blog

Information from Twitter's engineering team about our technology, tools and events.

Posts from Engineering: open source

Simplify Service Dependencies with Nodes

A history of how we handle service dependency graphs in search backend, and introducing a new library to make the implementation of asynchronous services in Java easier.

Read more...

Reinforcement Learning for Torch: Introducing torch-twrl

We are exploring RL as a learning paradigm, and to that end, Twitter Cortex built a framework for RL development. Today, Twitter is open sourcing torch-twrl to the world.

Read more...

Open Sourcing Twitter Heron

We’re open sourcing Heron, our real-time stream processing engine which has been powering all of Twitter’s real-time analytics for over two years.

Read more...

The release of Pants 1.0

After five years of development, open source project Pants 1.0.0 is here and we can’t wait for you to check it out.

Read more...

Distributed learning in Torch

We recently released Autograd for Torch, which greatly simplified our workflow when experimenting with complex deep learning architectures. The Twitter Cortex team is continuously investing in better tooling for manipulating our large datasets, and distributing training processes across machines in our cluster.

Today we’re open-sourcing four components of our training pipeline, so the community using Torch and/or Autograd can simplify their workflows when it comes to parallelizing training, and manipulating large, distributed datasets.

Read more...

Finatra 2.0: the fast, testable Scala services framework that powers Twitter

Introducing Finatra 2.0: a high-performance, scalable, testable framework powering production services at Twitter.

Read more...

Autograd for Torch

Simplifying neural nets with autograd for Torch.

Read more...

Hadoop filesystem at Twitter

Details about our HDFS deployments: HA, Federation, and ViewFs.

Read more...

Diffy: Testing services without writing tests

Diffy is an open-source tool that automatically detects potential bugs in HTTP-based services.

Read more...

Twitter at @MesosCon 2015

Once again, we’re pleased to sponsor and participate in #MesosCon. As heavy users of both Mesos and Apache Aurora to power our cloud infrastructure, we’re excited to be part of this growing community event.

Read more...

Pages