The Twitter Engineering Blog

Information from Twitter's engineering team about our technology, tools and events.

Posts from Engineering: experiments

Power, minimal detectable effect, and bucket size estimation in A/B tests

Figuring out the minimal number of users one must expose to an experimental treatment to collect actionable data is not a trivial task. We explain how we approach this problem with Twitter’s A/B testing platform (DDG), and how we communicate issues of statistical power to experimenters.

Read more...

Implications of use of multiple controls in an A/B test

Using a second control can be a tempting method of validating experiment results. We explore the statistics underlying usage of a second control, and conclude that this approach is strictly inferior to using a single large control.

Read more...

Detecting and avoiding bucket imbalance in A/B tests

Some simple techniques to detect potentially biased implementations of A/B tests.

Read more...

Twitter experimentation: technical overview

A description of Twitter’s A/B experiment evaluation pipeline and metric computation.

Read more...

The what and why of product experimentation at Twitter

We introduce the experimentation cycle, and discuss the role experimentation plays in decision-making and innovation.

Read more...