Improving communication across distributed teams

As a global company, Twitter has offices in numerous locations around the world. Optimizing collaboration and improving productivity between employees across offices is a challenge faced by many companies and something we’ve spent a great deal of time on. For our part, we’ve learned a few lessons about what it takes to make distributed engineering offices successful. Over the next few months, we’ll share what we’ve learned. We’ll also periodically offer workshops about how to do distributed engineering well. The first of these will be on Sept. 24 at General Assembly in San Francisco (shameless plug!).

In this post, we’ll describe the value, modes, and importance of communications between employees in different offices. We’ll share some data about how we do things, describe some of the common pitfalls that we identified, and how we’ve improved efficiency by addressing them.

In a TechCrunch interview, Alex Roetter, our SVP of Engineering, recently said:

“We try to build teams that thrive in an environment with a clear direction, but the details are unspecified. Teams are able to try a bunch of things and fail fast, and hold themselves accountable and measure themselves rigorously. We find that smaller teams are faster…we try to build them in that sort of environment.”

This kind of rapid iteration with small teams that Alex describes requires clear and constant communication between engineers, product managers, designers, researchers, and other functions. When everyone is in the same location, this is a lot easier. “Water-cooler” conversations or chats at offsites can lead to novel ideas, or foster cross-team collaboration. When employees work remotely, we have to do more work to ensure that they are able to have the kind of ad hoc conversations necessary to allow for innovation and ensure that different groups stay in sync.

Building teams that are as geographically co-located as possible is one easy way to address some of this. In a future post, our New York Engineering Operations Manager Valerie Kuo (@vyk) will talk about the ways in which Twitter organizes teams against various constraints, and how we deal with them.

In a modern-day technology company like Twitter, how often do meetings span locations? When I first started working here about a year ago, I was really curious about this – but it turns out relevant data are hard to find. To understand how pervasive cross-office collaboration really was, I analyzed data on every meeting held at Twitter between May 1, 2015 and July 25, 2015. 

There are a number of interesting conclusions:

• Over 20% of all meetings company-wide involve more than one city.
• Nearly 40% of meetings originating in our San Francisco HQ involve more than one city.
• Some cities work very closely together due to shared resources, strategies, market needs. For instance, Tokyo, Singapore, Seoul, and Sydney interact with each other far more than with they do with San Francisco.
• Despite the fact that most meetings are held in one city, meetings with multiple cities generally have more participants; so that 50% of meeting time at Twitter is spent in multi-city meetings.
• If we could increase the efficiency of multi-city meetings by 10%, we would save about 5 weeks of aggregate work every day.
• These data only include meetings that take place in conference rooms; since ad hoc video conferences between co-workers are excluded, we are almost certainly underestimating the scale of remote collaboration.

Given how critical interoffice communication is to a smooth operation, it’s clear that having a videoconferencing (VC) infrastructure is key to productivity. Recent upgrades to our VC systems have made this kind of collaboration substantially easier. The rest of this post will describe where we were a year ago, how we evaluated systems, and how changes we’ve made improve the working life of our employees.

How we started

A year ago, most Twitter HQ conference rooms were not outfitted with VC equipment. In distributed offices, penetration was higher, though by no means widespread. At that time, two different types of VC systems were deployed:

• Full VC rooms: These had enterprise-level VC systems including a hardware codec, table mics, built-in speakers and a touch screen controller. These rooms were fairly easy to use but relatively expensive to build. 
• VC-light rooms: These were outfitted with an LCD TV, a USB camera, and a USB speaker speaker/mic pod on the table. Calls were driven by the user’s laptop via cables to connect to VC calls. These rooms were very cost efficient, but also required the user to dedicate their laptop to the call.

We primarily used a SaaS service that provides video conferencing from a number of devices and allows interconnectivity to a variety of established VC systems. To use this system:

• The host or meeting organizer was responsible for creating a meeting in the app.
• The host then had to paste dial-in information into the meeting invite.
• In Full-VC rooms, users entered a 12-digit number into a the touch panel controller.
• In VC-light rooms, users connected their laptops to USB connectors, clicked on the invite, logged in to the service, then edited their audio and video sources.
• Users could also use mobile apps or the website, or dial in using a phone number.

Although we used this system extensively, there were some drawbacks: it regularly took over 5 minutes to initiate a VC connection. For a 30-minute meeting, this was over 15% of the meeting’s duration! And the quality of videoconferencing was poor because of several factors, including poor egress network bandwidth from our old New York engineering office, packet loss on the network, and the app’s prioritization of video quality over audio quality.

After noting these issues, I worked with key members of our IT team to make things better. We proposed these requirements to fix multi-city meetings:

1. 80%+ of rooms should be integrated with VC equipment in all offices.
2. Our calendaring system (Google Calendar) should be seamlessly integrated with video conferencing; i.e., VC equipment in each room should know the room’s schedule.
3. One-button entry into a video conference upon entering the meeting room.
4. Support for third-party phone dial-in.
5. Ability to join calls without a Google account.
6. Ability to support large number of dial-ins.

After a number of pilots, our IT team decided to go with a company-wide deployment of Google Hangouts plus Chrome for Meetings (CfM) boxes in conference rooms.

Deployment of Google Hangouts and CfM in a company that already uses Google Apps, as we do, makes adoption much easier — Google Calendar integration is built into the hardware. CfM boxes are not complicated, and can be connected to an LCD screen or integrated into full room systems. Our standard Hangouts room has an LCD screen mounted on the wall and Google’s standard mic/speaker pod and camera. A local HDMI connection with auto-input switching is included to enable low-friction presentations for local meetings.

Currently, CfM does not support 20+ person rooms. Since support for these rooms is important for larger meetings, we worked with AV vendors to develop a way to integrate higher-fidelity cameras and microphones typically found in these larger rooms. We’ve made it a standard across our offices, installing it in over 100 rooms globally. We will describe our approach in greater detail in a future post.

At this point, we have over 570 rooms set up with CfM, and most VC meetings are conducted using Hangouts and CfM. We’ve easily saved 3-5 minutes per meeting in VC setup time, and people are notably happier. Unfortunately, CfM still does not support (4), (5), and (6), and so we have maintained a few rooms to be compatible with our legacy setup. We hope that these issues will be addressed soon.

A Twitter conference room. Photo courtesy Alex Stillings.

Video conferencing is one of the most important pieces of internal infrastructure at Twitter, and CfM does its job well. While this setup is massive strides ahead of our initial deployment, I still fly to San Francisco once a month for in-person meetings, because the best VC available at a reasonable price doesn’t come close to substituting for in-person conversations. In larger meetings, being remote still feels a bit too much like watching a meeting on TV. And, we often have to mute mics when we’re not speaking and have periodics blips in A/V quality.

The more we reduce the cost and pain of remote collaboration, the more efficiently we – or any company – can run. We’re already a highly distributed company that takes advantage of a great deal of cross-office collaboration. Further improvements in VC technology will really unlock our ability to efficiently interact with the best employees – current and future – wherever they might be.