Twitter has hired another amazing engineer (and just in time, too). Robey came in over the weekend along with the rest of the eng and ops team to work on bugs—especially those that might be related to the case of the missing updates. In fact, Robey discovered what looks to have been the key: a timeout in Starling, the message queue server we recently open sourced.
Robey and Britt fixed the timeout and we’re watching now as messages find their proper destinations. However, we’ll wait to officially declare victory until this continues for a convincing amount of time and the team has had some rest. Good work Robey, and welcome to Twitter.