Man down!
Around 11 am in San Francisco, our main database db006, crashed because of too many connections. We have to put the service into an unscheduled maintenance mode to recover. Folks will see degraded service for the next few hours.
We'll keep you updated here and in Satisfaction.
220 Comments:
Darn, I just hit the wrong button in Blogger and deleted the first three comments on this post. But I still have them in my email. Here's what they said:
Mark Anderson said:
"Grrrrr..."
Jemimus said:
"Good luck guys, I hope you can recover from this without it having go down on you again immediately.
I don't know how hard it is to cluster and load-balance databases, but I am sure you are working in that direction."
Anonymous said:
"Does this have anything to do with RoR being unstable?"
@malbiniak said:
"/me hugs db006 and whispers reassuring nothings into their ear.
there there, there there."
I guess that explains why my damned badge wouldn't install. I thought it was me.
Goodluck, people. What kind of database are you using? Inquiring minds want to know.
Get well soon, twitter!
Luckily, all DBs go to heaven.
twitter rocks, but not the availability...
Better degraded and some communication from you guys about it than the sudden disappearances we've been experiencing lately.
Ok
What again?
What could be doing all of this? I read something about Jabber (last crash?), but I still can't see how Jabber could be related.
Good luck guys, and thanks for posting an update. I really like twitter, and I'd like it to succeed. I guess the fact that I'm upset when the service goes down, tells you that it's being used.
This DB crash is also causing other applications like Twhirl to crash too? I've been using web for the last 5 minutes and everything seems to go ok, but Twhirl doesn't retrieve absolutely nothing (and I'm sure the 70 tweets limit was not passed, there's not much people twittering in my timeline)
I hope you guys can get things back to work well soon.
@Biz, why can't twitter just purchase the equipment and get the OC-12s they need? Didn't you folks just raise $15M in funding?
I strongly suggest twitter sets up geo-located Co-Lo's around the world and the states.
Basically, West Coast, Midwest, East Coast locations. Then make the twitter servers round-robin from their localized server to other ones. So at least outages can be isolated to regions instead of the whole service.
It's frustrating that when I finally give Twitter a chance, your servers bite the dust.
Just some thoughts.
~Joe
Degraded relative to what? I've had mixed success connecting all week.
Feel better soon, twit! I may have some leftover cough drops from when I was sick...
;-)
¿No conocéis Eurovisión? Ese es el problema hoy!
Thanks for being so transparent!
Twitter is free - I miss it when it's down but cut 'em some slack....
Ah well, at least you'll have free publicity on hundreds of blogs across the globe now from people who wanted to post to Twitter but couldn't.
Even I posted about the downtime on my blog. :P
Just keep us up to date, and we can deal with it.
Bummer...we are all addicted here in Nashville...put a separate server for us so we can feed our addictions!
Also, would love it if Mobile Twitter had the @replies function like the web.
My guess: Crash caused by the Eurovision Song Contest… ;0)
I didn't do it on purpose and I won't do it again, I swear. Sorry about that.
Aw! All the best for a speedy recovery!!
aww twitter. i hope you get better soon. i love you twitter bird!
much love,
Mostly Lia
Hope you guys get stuff up and working soon.
And to all the anonymous people complaining, Its a free service you do not pay anything for.....
Quit your whining.
get well soon db006 :-)!
@gerdanperox: it's happening to Spaz too. Some sort of mistrapped error, I'm sure.
Hey folks, scaling up to massive increased activity is tough and you can't just buy a new server and roll it in like magic in a few minutes.
Of course if you're too tweaked you could always ask for your money back. *cough*
I trust they'll get it worked out. Having your cool site not working is not fun and anyone in this position wants to get back to stability as soon as possible.
Hang in there, Twitter folks!
So here's why my experimental client in python isn't working...it's not my fault :)
Get better soon, twitter db!
i think ill quit twitter if all these problems arent fixed! i love twitter, its great! but becoming very inconvenient
So that's what it was.
Good luck!
if i were an investor, i'd be asking to see your tech department's letters of resignation right about now.
Twitter doesn't work the eurovision night, that is awful! we cannot comment it!!!
Thank you
Thank you
Thank you
You guys are doing a great job. Twitter is amazing and thank you for keeping it free!
ok, I'm a new twitterbug today- does this happen often?
great.
please please please put it back on line soon. I woke up just to twitter. :( but its not working fine.
Twitter is in coma.. no twitter for few hours.
I hope the Tech Docs are listening and will ensure that Twitter Survive... :)
Time to just go to Wall Street and hire a real database person from Bloomberg or Reuters - someone that knows how to move massive amounts of data in short-form.
It's obvious that the SF/SV engineers are not up to the task. Hire someone that is up to the task and get this shit done.
thanks for keeping us posted, Biz. Good luck.
Thanks for keeping us posted Biz - good luck!
Enough already. How about monetize this thing? Some of us would actually pay for a working Twitter.
"We have added a consequence for failure."
Always glad you guys can recover fast; thanks again!
Oh.. man!
I love twitter!!!!!!!
dear twitter. i've had enough of your on again / off again attitude. i'm leaving you for friendfeed. all the cool kids are making the switch. sorry. you fail.
Sorry to hear about the problems. Twitter is so awesome that we get frustrated when it's down!
@biz and others have you thought about closing new sign-ups until things are a bit more stable? Might help with existing activity within twitter where the big complainers reside. :-)
I don't envy your position right now...keep on keeping on. And remember there are men/women dealing with harder things than server crashes around the world serving our country...especially important given it's Memorial Weekend.
Aruni
I'm about through w/ Twitter. One would think with 15m in funding they could buy some decent servers. Maybe talk to Google and get some input from them... Google.com never seems to go down from too many connections.
Cheers for the explanation Twitter Gods.
I'm rootin for you guys. I know it's been a hard week for ya'll. Hope you're getting some sleep. xo
yeah it has been a on and off week hope you get it fixed soon.
scaling at this growth rate can be really hard. good luck guys!
For the meantime there is whentwitterisdown
There's only one thing that can causes a computer to crash: a software bug, or also hardware failure ... There are only two things... ;^)
I wish people would stop spreading the meme that overworking computers makes them crash. I hear it all the time: "i crashed, i must have been doing too many things at once" etc. That's almost as silly as Kirk killing a robot by presenting it with a paradox. Grrr, makes me want to scream.
I'd say overloading db006 didn't make it crash, overloading it exposed a bug that made it crash. Anyone who actually knows about database servers & isn't blowing smoke like I am, please correct me if I'm mistaken. But I say, any software that builds in crashes as an expected failure mode, that's called buggy software. That's just the way I was brought up. Am I wrong in this??
Anyway, excuse me now, I have to go edit this thought down to 140 characters
If there's one thing I'm learning about life, it's that the road to success is uphill, made of dirt, and filled with potholes. And in the meantime it's raining.
*Passing wet towelettes to Twitter* Keep on moving, you're not alone! :-)
I don't pay to vote, either, yet I'll certainly complain when that's flawed, too. But I'm not going to leave the country on account of it, and I'm not giving up on Twitter, either.
Hey,
I'd like to say that while the outages are extremely frustrating, Twitter is a GREAT service and I'm staying with you guys to the end!
I hope Twitter gets well soon and we can enjoy Twitter to its max.
Have a good day and hope Twitted gets well soon!
PS: The reduced functionality mode, while it is reduced, is better than no Twitter at all.
Dudes,
Break down and do a few things with the VC funding:
(1) Hire a systerm architect.
(2) Hire someone who has worked with large databases and rapid data transfer.
You could recruit for (2) at SLAC which, until Google, was moving the most data in the world the fastest. Maybe they still do. And, they just had a layoff!
I digress (and yes, I was at SLAC from 2002-2004; great people, great place.)
(3) If you haven't already, get rid of that silly toy known as Ruby on Rails. It's shite.
wait...
I love all these 'armchair developer' f*cks who think they know how to solve Twitter's problems:
"I don't know how hard it is to cluster and load-balance databases, but I am sure you are working in that direction."
"Does this have anything to do with RoR being unstable?"
"why can't twitter just purchase the equipment and get the OC-12s they need? Didn't you folks just raise $15M in funding?"
"Time to just go to Wall Street and hire a real database person from Bloomberg or Reuters - someone that knows how to move massive amounts of data in short-form."
"One would think with 15m in funding they could buy some decent servers. Maybe talk to Google and get some input from them... Google.com never seems to go down from too many connections."
I especially love the reference to Google. You guys think Google got to where it is today in 18 months? They've been around for TEN YEARS. Google had problems scaling back in the day - it's just that you guys were in preschool then and didn't know what a computer was.
Give Twitter a break. The type of problems they are having take time to fix. They are very specific and very technical. I'm willing to bet there are only a handful of people in the world 'qualified' to work on a scaling problem such as this. But you can't just hire more people. Go read the 'Mythical Man Month' - no matter how many women you have, it still takes 9 months to make a baby.
Alex did a great post a few days ago outlining the problems they are having. They are replacing pieces of their system one at a time. Doing that type of work is hard enough, but add scaling issues and a fast-growing service to the mix, and it becomes a nightmare.
I wish only the best for the guys at Twitter. Hang in there, you'll get through it. And I'm willing to bet most of us will stick with you. I know I will.
Phil
Well, as long as it's you and not me.
Hang in there guys - this is the birthing process of a Web 2.0 BRILLIANT application which is the victim of its own success. As a developer I feel your pain. :-)
Aren't Growing pains fun? Gratz on the twitter growth spurt!
glad to become a part of you when you are down - it can only get better !
I say shoot whoever is to blame and be done with it... Nice and easy...Simple and clean...
and during Eurovision!!! ACK!
Just another get well soon comment ^^; thank you very much for the communication - it doesn't go unnoticed and it's very nice to see. Hope Twitter grows out of its growing pains soon xP
Looking forward to things being up & running again soon......sticking w/ you through thick & thin :-)
But I need to tweet about Eurovision :D
Haha! The Eurovision song contest is so bad even Twitter can't stand it!
One thousand bonus points to Twitter for the massive improvement in communication over the last week. I look forward to you being able to say "We've just implemented improvement X" instead of "This time we fell over for reason Y". ;) Good fortune!
I'm still giggling at all the "instant experts" who think that the decision to give you $15m funding only 2 working days ago will cause high-reliability infrastructure to magically appear overnight. Clearly they know nothing, and should read about the Dunning-Kruger Effect!
Pleeeease fix Twitter! Or I'll cut you.
I completely, totally, 100% don't care. I love it when it's up. It's free. Keep on keepin' on.
Come on Twitter, im getting sick of all the crashes and outages!
the comments from immature "developers" are obvious here. They are the ones for whom everything is "easy" and for whom "just buy a new server" or "just hire someone from NYC cesspool is the answer.
Having worked in a highly transactional environment for many years, its not an easy thing to manage, and we didn't have NEARLY the WORLDWIDE transactions Twitter tries to manage.
Keep up the good work and ignore these complaining doofuses.
"Young man shakes fist at cloud ..."
Positively the most positive thought beams are being sent in your direction... can you feel the love? Get well soon Twitter!
I hope you recover without any loss of data... I love twitter
Thanks for keeping us updated - I agree with the others who've commented that people need to stop grousing and whinging about a free service, too.
Best of luck to you!
Thanks for the blog link, Biz. Makes a difference. Cheers from East Coast Canada. (We are patient here, and give beers and lobsters to friends with difficulties)
@isle
Good luck with getting Twitter back up. Oh, but then it'll probably crash again. :(
This pretty much sums up how Twitter's been working lately:
http://twitter.com/_evan/statuses/819151648
:-(
Hi Twitter Big Shots ;)
To everything
(Turn, turn, turn)
There is a season
(Turn, turn, turn)
And a time to every purpose
Under Heaven
A time to build up, a time to break down
A time to dance, a time to mourn
A time to cast away stones, a time to gather stones together
To everything
(Turn, turn, turn)
There is a season
(Turn, turn, turn)
And a time to every purpose
Under Heaven
Good luck and I'm sure you'll have this thing up and running smoothly again in no time.
Peace, Love, Understanding and Respect,
Marilyn
I still love you, Twitter.
It's hard to complain about the uptime since the service is free *and* doesn't serve me ads, but I'm a bit worried about Twitter hurting it's growth potential and marketability to investors with all the DB problems.
I have been trying since this morning
(about the PDT time that now I see it went DOWN) and now I am trying again.
This time I can at least get in, but when I click on older-- to see what tweets I have missed, I get a message at the top that due to>>blah, blah,<<
Are we back to capacity or not?!
A computer keyboard, in a moment of anger, turns a law-abiding citizen into a criminal.
@Ken - exactly what I was thinking.
If you want whinging rights about twitter being down, donate money to fund the service. Then you get to bitch. Whilst you're using a free service, they're doing you a favour, so shut it.
Sky is clear on a Sunday morning. Hope nothing catastrophic happens in any part of the planet. Or people would need to tune in to the news because twitter's down.
Wish everyone well, twitter guys included.
Honestly.. I think the fact that we must sadly be without twitter up and running.. helps us NEVER take this awesome application for granted.
Thanks for the hard work..
Get well soon
SwitchingGranny
Excellent communication about the outage and service problems. Thanks for your honesty.
What? Twitter is down. Oh my gosh, call the New York Times.. That NEVER Happens unless there is air to breath on the planet.
=( we need twitter on im, on my gtalk...'cause from the timeline is really bad and slow =(
This is so stressful ... hurry up!!!!
I <3 twitter
Hire the Plenty of Fish guy, he can do with 4 servers what match.com can only do with 500.
writing a comment in this post :-)
(good luck with the maintenance)
Thanks for the update. Folks who are stressing without their "@replies" just search your username in terraminds, tweetscan or summize. You can even nab RSS of the search output and sit tight without missing a breath. Wishing Twitter well, and also wishing the humans behind Twitter well. This hasn't been a fun week for you. But you make a lot of lives a lot better. Well, mine anyway. So hang in there.
Yay, everything is working correctly again!
Thank you for providing information on this. I think you'll find most of us pretty patient if you just keep us informed of what's going on.
Guys...don't go the Friendster way! I love Twitter a lot and would hate to see that!
Picked a holiday weekend in the US - timing is good at least.
I've worked at a high-vis startup that had similar issues at times, so I know how it feels... scramble-n-fix.
As to all of the ideas posted here about solutions, let's go with the SETI@home distributed model, and at the same time also find the extra-terrestrials inhabiting the twittersphere.
hang in there :)
You guys can't close the new USV/Spark round fast enough, eh? Good luck :)
Hey, I've got it!
- All of you who know how to do it better, go do it.
- Those who are whining can go use the cool new service those nice folks are going to build for you for free.
- That will leave the rest of us happily on Twitter, and the overloading won't be such a problem.
Seriously, though... I sat on a patio all afternoon, enjoying a beautiful view and good food with friends, while Twitter's good people were probably tearing their hair out trying to get things working again.
And as Aruni pointed out, there are bigger problems in the world.
Good luck Twitter - we're rootin' for you.
well now i feel bad cause i just twittered about getting the IM back up...my bad haha.
i hate thisssssssssssssss!
I have never seen a tech company have so many failures and outages than Twitter. People will only put up with it for so long.
Major points to you guys for upping the communication with the community.
Gives some of us more to work with when we're defending the fact that you guys do an AMAZING job with everything - and put up a great service for us for free. :)
I keep getting the over capacity message. Is it really that bad?
Are the updates from mobile being affected as well?
I miss Twitter, so, so much, hope everything is ok, and thanks for all.
I just want to second the feeling of appreciation that you folks are adding more transparency on this here blog. Makes the outages more palatable.
I remember less than a year ago I couldn't understand Twitter's purpose for the everyday.
Although these growing pains have to be a bummer to you folks, isn't it so cool that peeps are tweeting like mad these days?
Congrats on your success! And here's to a speedy recovery. And vitamins. It's still growing after all.
Get well soon, sugar twits.
Man I preciate you're openly talking to us that way... Makes this services even funier and human. Good luck on that!
You guys have had phenomenal growth that must be a strain to keep up with. Good luck, because like most users, I love Titter. It is so nice to use, and it makes me feel good. GO TITTER!
Why don't you offer LOCAL channels (z.B. german language only) as a premium feature? Nobody can really read the "everyone" channel, reducing it to local would be realy great! I would pay for ist.
Plz guys make it work, we cant continue like this anymore, all the crashes.
i luv twitter stable.
make an effort
shimon
People, stop complaining!!
http://whentwitterisdown.com/
love the transparency and communication. LOVE. IT.
Go Twitter!
I really hope that you can get this sorted!
yup search function doesn't work.
Hmmm, seriously debating switching to something else like FriendFeed
Come on Twitter, find more technical staff and stable database servers!
Good luck. Remember all databases go to heaven!
Been there, done that. I am really sorry to hear you guys are not having a good time.
Everything is great when it is running fine but the real test of everyone is when it all hits the fan. Keep it cool and remember, it will all be over soon :D
Take care.
I can only imagine the psychological hell you guys are going through. Stay healthy!
New user. Love the platform. Keep it up!
The commoners don't understand that rewriting a service complicated enough to found a company on doesn't happen overnight, and redistributing a database across multiple servers doesn't happen overnight, and buying more brilliant engineers doesn't magically solve the problems any faster.
You are doing the right thing by placating the community with transparency and giving it your best.
Thanks guys!
Good work guys, hope that you can solve the problem! keep up the hard work!