<< Back to the main Twitter Blog page

Tuesday, May 20, 2008

Not True

Downtime is not good. We caused a database to fail during a routine update early this afternoon. We switched to a replica and expect this recovery to take place quickly. We're all working on it and watching right now as Twitter gets back up to speed. We have a thread open on our support forum which we'll update when we have more details to share. Getting our act together is something we continue to work on as we grow our company and our service.

28 Comments:

OpenID darnellclayton.com said...

Thanks Biz for letting us all know.

I started wondering after both my iPhone and PC both were reporting errors.

It was a sad five minuets. :-(

Either way, I at least have FriendFeed, so life is not too bad. :-)

5/20/08 2:04 PM  
Anonymous Manan said...

You guys seem to be way to unreliable with constant downtimes, it was 500s first then something else now this.

5/20/08 2:05 PM  
OpenID stilist said...

Having any sort of ‘crap, we broke it’ status page would be far superior than simply letting the site time out. Unexpected downtime can at least be mitigated by not being completely unhelpful to users.

5/20/08 2:17 PM  
Blogger Robert said...

Not looking good now - just got this error message:

Something is technically wrong.
Thanks for noticing—we're going to fix it up and have things back to normal soon.

I remember why I stopped logging in now :-(

5/20/08 3:00 PM  
OpenID geekmommy said...

Thanks for the explanation. Far better knowing what the root cause is, because it lets most of us know what a reasonable ETA for a fix is - rather than sitting in the dark wondering if it will be down 5 mins or 5 hours...

5/20/08 3:02 PM  
Anonymous HIghTechDad said...

Wow, this was fun.
How about diversifying your hosting or have a failover site hosted with someone like GoGrid?

Here's an idea for some smart developer, set up twitterqueue.com that queues up (and slowly releases) tweets when there is an outage so that there isn't a deluge of tweets coming in.

-HTD

5/20/08 3:30 PM  
Anonymous foodfight said...

Thank you for the update!

5/20/08 3:33 PM  
Blogger Jesse Stay said...

Communication is good, but why are you doing updates in the middle of the afternoon anyway? And why didn't you communicate before the update was made? These are common things most CTO's know and enforce well - your CTO is well over 3 strikes now if you ask me.

5/20/08 3:56 PM  
Anonymous Todd said...

Can you set something up so badges on external sites say something clever, instead of being blank, when things go boom?

5/20/08 5:02 PM  
Anonymous David Wallace said...

Twitter - we need to talk: http://www.searchrank.com/blog/2008/05/twitter-we-need-to-talk.html

5/20/08 6:40 PM  
Anonymous LiveCrunch said...

Guys i know the pain dont worry about cry babies that cant live without twitter more then5 minutes!!!! There is 50 people that cry about and millions of people that like twitter

5/20/08 7:00 PM  
Blogger Christin said...

Why don't y'all set up a donation page if you need help buying/fixing equipment that keeps Twitter up and running?
I know y'all are hoping for a buyout, but I'm sure not looking forward to adverts here....

5/20/08 8:00 PM  
Blogger tannerherriott said...

sucks that stuff went down today, but I'm still one of the millions who absolutely love twitter.

5/20/08 9:27 PM  
Blogger NathanaelB said...

Hope you fix your system stability issues real soon - particularly as I have an idea for a Twitter app but as it's about personal safety and security I can't risk having it based on infrastructure that's unreliable. But in the meantime we always have the #twitter IRC channel on freenode as our back-up plan so we're good for now.

5/20/08 11:18 PM  
Blogger Daniel said...

You should only do HUGE UNUSUAL updates since the "routine" ones seem to always be causing the problems.

5/21/08 12:00 AM  
Blogger xnera said...

An external status page would be awesome.

5/21/08 8:09 AM  
Anonymous Anonymous said...

Yeah, it would be great if you guys chose sometime other than the middle of the afternoon for updates. How about late at night or on the weekend, like pretty much every other service in the world?

Mucking with production servers in the middle of the day is not to be taken lightly. Yet, you guys keep doing it and keep messing up. Getting your act together would seem to imply learning from your mistakes.

5/21/08 8:28 AM  
Blogger the.blogm said...

I'm glad yall were able to fix up the glitch. Thanks crew

5/21/08 8:48 AM  
Anonymous Mark said...

You have VC, how hard is it to implement an Oracle 10g grid database? Your db will never go offline ever again.

5/21/08 9:21 AM  
Blogger neil said...

I know a TERRIFIC Oracle guy :D. I even wrote a plsql interface to twitter to send/view tweets out of an Oracle stored procedure.

5/21/08 9:49 AM  
Blogger Steve Streza said...

If what you're saying is true, Biz, then you guys really need to put some better failover redundancy into the system. One db getting brought down takes out the entire service for an extended period of time?

5/21/08 1:25 PM  
Blogger Kyle said...

So, what's the excuse today?

5/21/08 1:59 PM  
Blogger Mike said...

Twitter is down again at 5pm EST on Wednesday. It would be more useful if you could post an entry to the weblog each time the website is actually up.

5/21/08 2:02 PM  
Anonymous foodfight said...

We're back off-line again? :(

5/21/08 2:40 PM  
Blogger NathanaelB said...

@Anonymous The outage was the perfect time for us Aussies - between 5pm and 7pm when we're driving home from work and having dinner!

However the outage in early April right in the middle of BarCampCanberra when we were using Twitter to microblog and post presentation schedules etc with over 70 tweets throughout the day ... that outage wasn't quite so well timed ...

5/21/08 2:41 PM  
Anonymous Anonymous said...

It's down AGAIN? This is really becoming absurd. You're one of the greatest Web 2.0 companies out there, but yet your reliability is awful. Try hosting with MediaTemple perhaps?? I appreciate that you're working on it, but, the downtime has been too much for too long lately.

5/21/08 4:42 PM  
Blogger Mario said...

I wonder how many people are switching over to Pownce now.

5/21/08 6:32 PM  
Anonymous Rob Bazinet said...

If it is a stunt then the guys at Twitter are truly stupid. Twitter is becoming so unreliable that I hope everyone goes to Pownce.

If everyone I followed went over, I would dump Twitter for good.

5/21/08 6:40 PM  

Post a Comment

Links to this post:

Create a Link

<< Home