Weekend Web Weirdness
On Friday, we successfully deployed a new memcache project as part of our overall work to create a more scalable service. After the deploy, we needed to move a lot of data with minimum impact on service quality. To do this, we put together some code that moves data only as it is requested.
This process kept a minimum on service disruption but did cause Twitter to have a complex conversation with two sets of caches over the weekend and into today. This resulted in some caching issues—namely, the /home timeline cache wasn't being updated correctly for everyone.
We're aware of this, we realize that it's annoying, and we're meeting today about how to best finish up this project and clean up any remaining bugs. Thanks to everyone who checked in with us on Satisfaction, @replies, and email over the weekend. Overall, completing this memcache project is a big win that will lead to increased stability.
Update: We're working on this project more today and will be checking in again this evening. Also, we keep folks updated as much as possible on our forum over at Get Satisfaction (in case you're only visiting this blog for news).
Update: We've deployed code that gets Twitter talking to one - and only one - pool
of memcache servers. This eliminates a lot of potential confusion and is progress. But we are still noticing related symptoms so we're still investigating and making improvements.
Update: Okay, here's a more satisfying update. We're catching up on things and backfilling the timelines. Once things are caught up, we should be good—and successfully moved to the new memcaching scheme.
This process kept a minimum on service disruption but did cause Twitter to have a complex conversation with two sets of caches over the weekend and into today. This resulted in some caching issues—namely, the /home timeline cache wasn't being updated correctly for everyone.
We're aware of this, we realize that it's annoying, and we're meeting today about how to best finish up this project and clean up any remaining bugs. Thanks to everyone who checked in with us on Satisfaction, @replies, and email over the weekend. Overall, completing this memcache project is a big win that will lead to increased stability.
Update: We're working on this project more today and will be checking in again this evening. Also, we keep folks updated as much as possible on our forum over at Get Satisfaction (in case you're only visiting this blog for news).
Update: We've deployed code that gets Twitter talking to one - and only one - pool
of memcache servers. This eliminates a lot of potential confusion and is progress. But we are still noticing related symptoms so we're still investigating and making improvements.
Update: Okay, here's a more satisfying update. We're catching up on things and backfilling the timelines. Once things are caught up, we should be good—and successfully moved to the new memcaching scheme.
posted by Biz at 9:27 AM
13 Comments:
If you want to fix this issue, you can follow these directions.
"we realize it's annoying" - ?
When the whole point of the service (to post messages and make them available) isn't working, it's not just annoying, it's broken.
Why, when it's been broken for 3 days, is this the first communication about the problem?
Thnx, I would love if you could share some experiences on what you did with the new memcache project.
I'm glad there's finally some official word from Twitter about this.
-Adam
So... you pushed out a FUNCTIONAL CHANGE... on a FRIDAY NIGHT?? What braindead IT person does that? Unless you have all the staff around on the weekend you never push functional updates on a Friday for exactly this reason.
Then you basically went dark on us over the weekend. A company that facilitates communication... not doing it. Screwups happen (even if they shouldn't) but there are so MANY examples of how to properly handle them (hint... communication is good) that you don't get a pass here.
Oh and it's a big win? Then why are a lot of us not seeing updates? How is it a win if the service DOESN'T WORK FOR US?
No one's perfect, but pushing changes on a Friday and not communicating for 48 hours are two textbook examples of what NOT to do. Blowing smoke into nether regions doesn't actually help, it's just offensive.
hey biz,
once you have met, can you give us an estimate of how long it will take (re-caching or whatever is solution)?
i've begun to use twitter as my main form of communication (even over email).
thanks, and i know it's a ton of work to do what you are doing.. we just luv twitter and wince when it's less there for us. :)
mary
Thanks a bunch for the update! It means the world to all of us.
I dislike seeing all the whiners that complain nonstop when these things happen. Just like it rains outside, these things /do/ happen.
Stop complaining twitter-ers! It's going to make everyone LESS inclined to help you!
Hey Biz,
Thanks for the explanation.
Given your current understanding of the issue, can you provide an estimate on when things will be back to normal?
Uh oh... Comment moderation has been enabled. All comments must be approved by the blog author.
How the heck am I @acomputerpro going to get my comment posted? I better quit while I'm ahead. Hi @Biz. Cool name, btw. Did I ever mention how much I love Twitter and what a great service you provide? Keep up the good work ;)
Thanks so much for the update. Great to see some progress. Your hard work is VERY much appreciated.
You have a problem Twitter and what's even worse, it doesn't look like you're listening.
Sure you were trying to improve experiences, but you made it worse.
Sure you were trying to fix the problem, but you didn't even communicate (a) what the problem was and (b) what you intended to do about it.
Come on guys, I know you want to be secretive but secretive didn't work so well for Linden Labs and it won't work so well here either.
I stopped using Twitter this weekend. That is a VERY big deal. It wasn't valuable to me or my time because it didn't work.
Start talking, start having a conversation when the problems are happening.
Because really, I rely on Twitter, but it's becoming unreliable.
hi, y do i c @ replies not addressed me? this is annoying. please turn it off. previously it showed only replies to me..
-ravishankar
Post a Comment
Links to this post:
Create a Link
<< Home