<< Back to the main Twitter Blog page

Saturday, May 24, 2008

Man down!

Around 11 am in San Francisco, our main database db006, crashed because of too many connections. We have to put the service into an unscheduled maintenance mode to recover. Folks will see degraded service for the next few hours.

We'll keep you updated here and in Satisfaction.

220 Comments:

Blogger Biz said...

Darn, I just hit the wrong button in Blogger and deleted the first three comments on this post. But I still have them in my email. Here's what they said:

Mark Anderson said:

"Grrrrr..."

Jemimus said:

"Good luck guys, I hope you can recover from this without it having go down on you again immediately.

I don't know how hard it is to cluster and load-balance databases, but I am sure you are working in that direction."

Anonymous said:

"Does this have anything to do with RoR being unstable?"

@malbiniak said:

"/me hugs db006 and whispers reassuring nothings into their ear.

there there, there there."

5/24/08 12:16 PM  
Anonymous bc said...

I guess that explains why my damned badge wouldn't install. I thought it was me.
Goodluck, people. What kind of database are you using? Inquiring minds want to know.

5/24/08 12:25 PM  
Anonymous Kara said...

Get well soon, twitter!

5/24/08 12:55 PM  
Blogger Scott Fleckenstein said...

Luckily, all DBs go to heaven.

5/24/08 12:59 PM  
Blogger Jamal Mavadat said...

twitter rocks, but not the availability...

5/24/08 1:04 PM  
OpenID markstoneman said...

Better degraded and some communication from you guys about it than the sudden disappearances we've been experiencing lately.

5/24/08 1:06 PM  
Blogger Олег said...

Ok

5/24/08 1:09 PM  
Anonymous Lawrence said...

What again?

What could be doing all of this? I read something about Jabber (last crash?), but I still can't see how Jabber could be related.

5/24/08 1:09 PM  
Blogger lance said...

Good luck guys, and thanks for posting an update. I really like twitter, and I'd like it to succeed. I guess the fact that I'm upset when the service goes down, tells you that it's being used.

5/24/08 1:10 PM  
Blogger Gerdanperox said...

This DB crash is also causing other applications like Twhirl to crash too? I've been using web for the last 5 minutes and everything seems to go ok, but Twhirl doesn't retrieve absolutely nothing (and I'm sure the 70 tweets limit was not passed, there's not much people twittering in my timeline)

I hope you guys can get things back to work well soon.

5/24/08 1:10 PM  
Anonymous Joe Manna said...

@Biz, why can't twitter just purchase the equipment and get the OC-12s they need? Didn't you folks just raise $15M in funding?

I strongly suggest twitter sets up geo-located Co-Lo's around the world and the states.

Basically, West Coast, Midwest, East Coast locations. Then make the twitter servers round-robin from their localized server to other ones. So at least outages can be isolated to regions instead of the whole service.

It's frustrating that when I finally give Twitter a chance, your servers bite the dust.

Just some thoughts.

~Joe

5/24/08 1:11 PM  
Anonymous Anonymous said...

Degraded relative to what? I've had mixed success connecting all week.

5/24/08 1:16 PM  
Blogger Talia Davis - PR Director said...

Feel better soon, twit! I may have some leftover cough drops from when I was sick...

;-)

5/24/08 1:20 PM  
Blogger José Mª Martínez-Priego said...

¿No conocéis Eurovisión? Ese es el problema hoy!

5/24/08 1:27 PM  
Blogger Jake Jarvis said...

Thanks for being so transparent!

5/24/08 1:29 PM  
Anonymous Jay Greasley said...

Twitter is free - I miss it when it's down but cut 'em some slack....

5/24/08 1:33 PM  
OpenID Andrew said...

Ah well, at least you'll have free publicity on hundreds of blogs across the globe now from people who wanted to post to Twitter but couldn't.

Even I posted about the downtime on my blog. :P

5/24/08 1:33 PM  
Anonymous Anonymous said...

Just keep us up to date, and we can deal with it.

5/24/08 1:39 PM  
Blogger GingerSnaps said...

Bummer...we are all addicted here in Nashville...put a separate server for us so we can feed our addictions!

Also, would love it if Mobile Twitter had the @replies function like the web.

5/24/08 1:43 PM  
OpenID nwinton said...

My guess: Crash caused by the Eurovision Song Contest… ;0)

5/24/08 1:49 PM  
Blogger DieLaughing said...

I didn't do it on purpose and I won't do it again, I swear. Sorry about that.

5/24/08 1:49 PM  
Anonymous Lynn said...

Aw! All the best for a speedy recovery!!

5/24/08 1:52 PM  
Blogger Mostly Lisa said...

aww twitter. i hope you get better soon. i love you twitter bird!

much love,
Mostly Lia

5/24/08 1:57 PM  
Blogger Ken said...

Hope you guys get stuff up and working soon.

And to all the anonymous people complaining, Its a free service you do not pay anything for.....

Quit your whining.

5/24/08 2:00 PM  
Anonymous Jan said...

get well soon db006 :-)!

5/24/08 2:00 PM  
Blogger jkc said...

@gerdanperox: it's happening to Spaz too. Some sort of mistrapped error, I'm sure.

5/24/08 2:00 PM  
Blogger MetaGrrrl said...

Hey folks, scaling up to massive increased activity is tough and you can't just buy a new server and roll it in like magic in a few minutes.

Of course if you're too tweaked you could always ask for your money back. *cough*

I trust they'll get it worked out. Having your cool site not working is not fun and anyone in this position wants to get back to stability as soon as possible.

Hang in there, Twitter folks!

5/24/08 2:01 PM  
Blogger Raibaz said...

So here's why my experimental client in python isn't working...it's not my fault :)

Get better soon, twitter db!

5/24/08 2:03 PM  
Anonymous Priscilla said...

i think ill quit twitter if all these problems arent fixed! i love twitter, its great! but becoming very inconvenient

5/24/08 2:03 PM  
Blogger Anna said...

So that's what it was.
Good luck!

5/24/08 2:04 PM  
Anonymous david said...

if i were an investor, i'd be asking to see your tech department's letters of resignation right about now.

5/24/08 2:15 PM  
Blogger María said...

Twitter doesn't work the eurovision night, that is awful! we cannot comment it!!!

5/24/08 2:18 PM  
OpenID nbabyak said...

Thank you
Thank you
Thank you

5/24/08 2:18 PM  
Blogger Francisco said...

You guys are doing a great job. Twitter is amazing and thank you for keeping it free!

5/24/08 2:24 PM  
Anonymous Anonymous said...

ok, I'm a new twitterbug today- does this happen often?

5/24/08 2:24 PM  
Blogger ~PiScEaNaTeD~ said...

great.
please please please put it back on line soon. I woke up just to twitter. :( but its not working fine.

Twitter is in coma.. no twitter for few hours.

I hope the Tech Docs are listening and will ensure that Twitter Survive... :)

5/24/08 2:29 PM  
Anonymous Anonymous said...

Time to just go to Wall Street and hire a real database person from Bloomberg or Reuters - someone that knows how to move massive amounts of data in short-form.

It's obvious that the SF/SV engineers are not up to the task. Hire someone that is up to the task and get this shit done.

5/24/08 2:34 PM  
Blogger mike said...

thanks for keeping us posted, Biz. Good luck.

5/24/08 2:42 PM  
Anonymous Mike Walsh said...

Thanks for keeping us posted Biz - good luck!

5/24/08 2:44 PM  
Anonymous Erazmus said...

Enough already. How about monetize this thing? Some of us would actually pay for a working Twitter.

5/24/08 2:45 PM  
Blogger Matt said...

"We have added a consequence for failure."

Always glad you guys can recover fast; thanks again!

5/24/08 2:49 PM  
OpenID nitosblog.com said...

Oh.. man!
I love twitter!!!!!!!

5/24/08 2:53 PM  
Anonymous Anonymous said...

dear twitter. i've had enough of your on again / off again attitude. i'm leaving you for friendfeed. all the cool kids are making the switch. sorry. you fail.

5/24/08 2:56 PM  
Blogger Aruni said...

Sorry to hear about the problems. Twitter is so awesome that we get frustrated when it's down!

@biz and others have you thought about closing new sign-ups until things are a bit more stable? Might help with existing activity within twitter where the big complainers reside. :-)

I don't envy your position right now...keep on keeping on. And remember there are men/women dealing with harder things than server crashes around the world serving our country...especially important given it's Memorial Weekend.

Aruni

5/24/08 3:02 PM  
Blogger Jack said...

I'm about through w/ Twitter. One would think with 15m in funding they could buy some decent servers. Maybe talk to Google and get some input from them... Google.com never seems to go down from too many connections.

5/24/08 3:06 PM  
Blogger PkmnTrainerJ said...

Cheers for the explanation Twitter Gods.

5/24/08 3:15 PM  
Blogger willo said...

I'm rootin for you guys. I know it's been a hard week for ya'll. Hope you're getting some sleep. xo

5/24/08 3:15 PM  
Anonymous syn said...

yeah it has been a on and off week hope you get it fixed soon.
scaling at this growth rate can be really hard. good luck guys!

For the meantime there is whentwitterisdown

5/24/08 3:21 PM  
Anonymous smallduck said...

There's only one thing that can causes a computer to crash: a software bug, or also hardware failure ... There are only two things... ;^)

I wish people would stop spreading the meme that overworking computers makes them crash. I hear it all the time: "i crashed, i must have been doing too many things at once" etc. That's almost as silly as Kirk killing a robot by presenting it with a paradox. Grrr, makes me want to scream.

I'd say overloading db006 didn't make it crash, overloading it exposed a bug that made it crash. Anyone who actually knows about database servers & isn't blowing smoke like I am, please correct me if I'm mistaken. But I say, any software that builds in crashes as an expected failure mode, that's called buggy software. That's just the way I was brought up. Am I wrong in this??

Anyway, excuse me now, I have to go edit this thought down to 140 characters

5/24/08 3:23 PM  
Blogger mousewords said...

If there's one thing I'm learning about life, it's that the road to success is uphill, made of dirt, and filled with potholes. And in the meantime it's raining.

*Passing wet towelettes to Twitter* Keep on moving, you're not alone! :-)

5/24/08 3:25 PM  
Blogger kiki said...

I don't pay to vote, either, yet I'll certainly complain when that's flawed, too. But I'm not going to leave the country on account of it, and I'm not giving up on Twitter, either.

5/24/08 3:27 PM  
Blogger Shaun R said...

Hey,
I'd like to say that while the outages are extremely frustrating, Twitter is a GREAT service and I'm staying with you guys to the end!
I hope Twitter gets well soon and we can enjoy Twitter to its max.
Have a good day and hope Twitted gets well soon!

PS: The reduced functionality mode, while it is reduced, is better than no Twitter at all.

5/24/08 3:29 PM  
Blogger The Mad Doctor said...

Dudes,
Break down and do a few things with the VC funding:

(1) Hire a systerm architect.
(2) Hire someone who has worked with large databases and rapid data transfer.

You could recruit for (2) at SLAC which, until Google, was moving the most data in the world the fastest. Maybe they still do. And, they just had a layoff!

I digress (and yes, I was at SLAC from 2002-2004; great people, great place.)

(3) If you haven't already, get rid of that silly toy known as Ruby on Rails. It's shite.

5/24/08 3:30 PM  
Blogger fspina said...

wait...

5/24/08 3:32 PM  
Anonymous Phil said...

I love all these 'armchair developer' f*cks who think they know how to solve Twitter's problems:

"I don't know how hard it is to cluster and load-balance databases, but I am sure you are working in that direction."

"Does this have anything to do with RoR being unstable?"

"why can't twitter just purchase the equipment and get the OC-12s they need? Didn't you folks just raise $15M in funding?"

"Time to just go to Wall Street and hire a real database person from Bloomberg or Reuters - someone that knows how to move massive amounts of data in short-form."

"One would think with 15m in funding they could buy some decent servers. Maybe talk to Google and get some input from them... Google.com never seems to go down from too many connections."

I especially love the reference to Google. You guys think Google got to where it is today in 18 months? They've been around for TEN YEARS. Google had problems scaling back in the day - it's just that you guys were in preschool then and didn't know what a computer was.

Give Twitter a break. The type of problems they are having take time to fix. They are very specific and very technical. I'm willing to bet there are only a handful of people in the world 'qualified' to work on a scaling problem such as this. But you can't just hire more people. Go read the 'Mythical Man Month' - no matter how many women you have, it still takes 9 months to make a baby.

Alex did a great post a few days ago outlining the problems they are having. They are replacing pieces of their system one at a time. Doing that type of work is hard enough, but add scaling issues and a fast-growing service to the mix, and it becomes a nightmare.

I wish only the best for the guys at Twitter. Hang in there, you'll get through it. And I'm willing to bet most of us will stick with you. I know I will.

Phil

5/24/08 3:34 PM  
Blogger Liz said...

Well, as long as it's you and not me.

5/24/08 3:37 PM  
OpenID punctuation said...

Hang in there guys - this is the birthing process of a Web 2.0 BRILLIANT application which is the victim of its own success. As a developer I feel your pain. :-)

5/24/08 3:38 PM  
Anonymous Jacked01 said...

Aren't Growing pains fun? Gratz on the twitter growth spurt!

5/24/08 3:44 PM  
Anonymous barryslounge said...

glad to become a part of you when you are down - it can only get better !

5/24/08 3:45 PM  
Blogger Kathleen Marie said...

I say shoot whoever is to blame and be done with it... Nice and easy...Simple and clean...

5/24/08 3:49 PM  
Anonymous misfitduck said...

and during Eurovision!!! ACK!

5/24/08 3:52 PM  
Blogger Keakealani said...

Just another get well soon comment ^^; thank you very much for the communication - it doesn't go unnoticed and it's very nice to see. Hope Twitter grows out of its growing pains soon xP

5/24/08 3:58 PM  
Anonymous Kate Olson said...

Looking forward to things being up & running again soon......sticking w/ you through thick & thin :-)

5/24/08 4:05 PM  
OpenID mikemacd said...

But I need to tweet about Eurovision :D

5/24/08 4:06 PM  
Blogger NXile said...

Haha! The Eurovision song contest is so bad even Twitter can't stand it!

5/24/08 4:19 PM  
Anonymous Stilgherrian said...

One thousand bonus points to Twitter for the massive improvement in communication over the last week. I look forward to you being able to say "We've just implemented improvement X" instead of "This time we fell over for reason Y". ;) Good fortune!

I'm still giggling at all the "instant experts" who think that the decision to give you $15m funding only 2 working days ago will cause high-reliability infrastructure to magically appear overnight. Clearly they know nothing, and should read about the Dunning-Kruger Effect!

5/24/08 4:20 PM  
Anonymous Ask A Chola said...

Pleeeease fix Twitter! Or I'll cut you.

5/24/08 4:23 PM  
Blogger Jeff said...

I completely, totally, 100% don't care. I love it when it's up. It's free. Keep on keepin' on.

5/24/08 4:32 PM  
Blogger Lazymonkey44 said...

Come on Twitter, im getting sick of all the crashes and outages!

5/24/08 4:41 PM  
Anonymous Anonymous said...

the comments from immature "developers" are obvious here. They are the ones for whom everything is "easy" and for whom "just buy a new server" or "just hire someone from NYC cesspool is the answer.

Having worked in a highly transactional environment for many years, its not an easy thing to manage, and we didn't have NEARLY the WORLDWIDE transactions Twitter tries to manage.

Keep up the good work and ignore these complaining doofuses.

5/24/08 4:46 PM  
OpenID jrae83160 said...

"Young man shakes fist at cloud ..."

5/24/08 4:50 PM  
Blogger Zen Runner said...

Positively the most positive thought beams are being sent in your direction... can you feel the love? Get well soon Twitter!

5/24/08 4:56 PM  
Anonymous mujra said...

I hope you recover without any loss of data... I love twitter

5/24/08 5:07 PM  
OpenID ciliandis said...

Thanks for keeping us updated - I agree with the others who've commented that people need to stop grousing and whinging about a free service, too.

Best of luck to you!

5/24/08 5:15 PM  
Blogger Islander said...

Thanks for the blog link, Biz. Makes a difference. Cheers from East Coast Canada. (We are patient here, and give beers and lobsters to friends with difficulties)
@isle

5/24/08 5:20 PM  
Anonymous Chris Thomson said...

Good luck with getting Twitter back up. Oh, but then it'll probably crash again. :(


This pretty much sums up how Twitter's been working lately:

http://twitter.com/_evan/statuses/819151648

:-(

5/24/08 5:24 PM  
Blogger Badthing1 said...

Hi Twitter Big Shots ;)

To everything
(Turn, turn, turn)
There is a season
(Turn, turn, turn)
And a time to every purpose
Under Heaven

A time to build up, a time to break down
A time to dance, a time to mourn
A time to cast away stones, a time to gather stones together

To everything
(Turn, turn, turn)
There is a season
(Turn, turn, turn)
And a time to every purpose
Under Heaven

Good luck and I'm sure you'll have this thing up and running smoothly again in no time.

Peace, Love, Understanding and Respect,
Marilyn

5/24/08 5:27 PM  
Blogger Tembrooke said...

I still love you, Twitter.

5/24/08 5:28 PM  
Anonymous Chris Hooley said...

It's hard to complain about the uptime since the service is free *and* doesn't serve me ads, but I'm a bit worried about Twitter hurting it's growth potential and marketability to investors with all the DB problems.

5/24/08 5:32 PM  
Blogger Mammo said...

I have been trying since this morning
(about the PDT time that now I see it went DOWN) and now I am trying again.

This time I can at least get in, but when I click on older-- to see what tweets I have missed, I get a message at the top that due to>>blah, blah,<<

Are we back to capacity or not?!

5/24/08 5:53 PM  
Anonymous Anonymous said...

A computer keyboard, in a moment of anger, turns a law-abiding citizen into a criminal.

5/24/08 5:56 PM  
OpenID xugglybug said...

@Ken - exactly what I was thinking.

If you want whinging rights about twitter being down, donate money to fund the service. Then you get to bitch. Whilst you're using a free service, they're doing you a favour, so shut it.

5/24/08 5:58 PM  
Blogger friarminor said...

Sky is clear on a Sunday morning. Hope nothing catastrophic happens in any part of the planet. Or people would need to tune in to the news because twitter's down.

Wish everyone well, twitter guys included.

5/24/08 6:15 PM  
Blogger Martha said...

Honestly.. I think the fact that we must sadly be without twitter up and running.. helps us NEVER take this awesome application for granted.

Thanks for the hard work..

Get well soon

SwitchingGranny

5/24/08 6:20 PM  
Blogger Karen G. Anderson said...

Excellent communication about the outage and service problems. Thanks for your honesty.

5/24/08 6:23 PM  
Blogger Thom Singer said...

What? Twitter is down. Oh my gosh, call the New York Times.. That NEVER Happens unless there is air to breath on the planet.

5/24/08 6:25 PM  
Blogger Katyca said...

=( we need twitter on im, on my gtalk...'cause from the timeline is really bad and slow =(

This is so stressful ... hurry up!!!!

I <3 twitter

5/24/08 6:26 PM  
Blogger Dugdale said...

Hire the Plenty of Fish guy, he can do with 4 servers what match.com can only do with 500.

5/24/08 6:35 PM  
Anonymous mollyparker said...

writing a comment in this post :-)
(good luck with the maintenance)

5/24/08 6:52 PM  
Anonymous Laura "Pistachio" Fitton said...

Thanks for the update. Folks who are stressing without their "@replies" just search your username in terraminds, tweetscan or summize. You can even nab RSS of the search output and sit tight without missing a breath. Wishing Twitter well, and also wishing the humans behind Twitter well. This hasn't been a fun week for you. But you make a lot of lives a lot better. Well, mine anyway. So hang in there.

5/24/08 7:02 PM  
OpenID twermund said...

Yay, everything is working correctly again!

5/24/08 7:30 PM  
Anonymous Scott said...

Thank you for providing information on this. I think you'll find most of us pretty patient if you just keep us informed of what's going on.

5/24/08 7:39 PM  
Blogger Narayan said...

Guys...don't go the Friendster way! I love Twitter a lot and would hate to see that!

5/24/08 8:01 PM  
Anonymous Chief Social Officer said...

Picked a holiday weekend in the US - timing is good at least.

I've worked at a high-vis startup that had similar issues at times, so I know how it feels... scramble-n-fix.

As to all of the ideas posted here about solutions, let's go with the SETI@home distributed model, and at the same time also find the extra-terrestrials inhabiting the twittersphere.

5/24/08 8:08 PM  
Anonymous drew olanoff said...

hang in there :)

5/24/08 8:18 PM  
Blogger Kyle S said...

You guys can't close the new USV/Spark round fast enough, eh? Good luck :)

5/24/08 8:36 PM  
Anonymous LindaEskin said...

Hey, I've got it!

- All of you who know how to do it better, go do it.
- Those who are whining can go use the cool new service those nice folks are going to build for you for free.
- That will leave the rest of us happily on Twitter, and the overloading won't be such a problem.

Seriously, though... I sat on a patio all afternoon, enjoying a beautiful view and good food with friends, while Twitter's good people were probably tearing their hair out trying to get things working again.

And as Aruni pointed out, there are bigger problems in the world.

Good luck Twitter - we're rootin' for you.

5/24/08 8:50 PM  
Blogger Kelly said...

well now i feel bad cause i just twittered about getting the IM back up...my bad haha.

5/24/08 9:11 PM  
Blogger zeze pascolato said...

i hate thisssssssssssssss!

5/24/08 9:17 PM  
Blogger J said...

I have never seen a tech company have so many failures and outages than Twitter. People will only put up with it for so long.

5/24/08 9:43 PM  
OpenID geekmommy said...

Major points to you guys for upping the communication with the community.
Gives some of us more to work with when we're defending the fact that you guys do an AMAZING job with everything - and put up a great service for us for free. :)

5/24/08 9:45 PM  
Blogger Joanna said...

I keep getting the over capacity message. Is it really that bad?

Are the updates from mobile being affected as well?

5/24/08 9:49 PM  
Blogger luciérnaga said...

I miss Twitter, so, so much, hope everything is ok, and thanks for all.

5/24/08 9:53 PM  
Blogger Webomatica said...

I just want to second the feeling of appreciation that you folks are adding more transparency on this here blog. Makes the outages more palatable.

5/24/08 9:55 PM  
Anonymous boo said...

I remember less than a year ago I couldn't understand Twitter's purpose for the everyday.

Although these growing pains have to be a bummer to you folks, isn't it so cool that peeps are tweeting like mad these days?

Congrats on your success! And here's to a speedy recovery. And vitamins. It's still growing after all.

5/24/08 10:03 PM  
Blogger Molly said...

Get well soon, sugar twits.

5/24/08 10:11 PM  
Blogger FroznBlue said...

Man I preciate you're openly talking to us that way... Makes this services even funier and human. Good luck on that!

5/24/08 10:12 PM  
Anonymous Anonymous said...

You guys have had phenomenal growth that must be a strain to keep up with. Good luck, because like most users, I love Titter. It is so nice to use, and it makes me feel good. GO TITTER!

5/24/08 10:36 PM  
Anonymous Chris said...

Why don't you offer LOCAL channels (z.B. german language only) as a premium feature? Nobody can really read the "everyone" channel, reducing it to local would be realy great! I would pay for ist.

5/24/08 11:12 PM  
Blogger שמעון טהר said...

Plz guys make it work, we cant continue like this anymore, all the crashes.
i luv twitter stable.

make an effort
shimon

5/24/08 11:28 PM  
Anonymous jyoseph said...

People, stop complaining!!

http://whentwitterisdown.com/

5/24/08 11:46 PM  
Blogger islaygirl said...

love the transparency and communication. LOVE. IT.

Go Twitter!

5/25/08 12:07 AM  
Blogger Walter Pike said...

I really hope that you can get this sorted!

5/25/08 12:08 AM  
Anonymous Samuel Lavoie said...

yup search function doesn't work.

5/25/08 12:15 AM  
Anonymous Kurt said...

Hmmm, seriously debating switching to something else like FriendFeed

5/25/08 12:26 AM  
Anonymous Wanna said...

Come on Twitter, find more technical staff and stable database servers!

5/25/08 12:45 AM  
OpenID challgren said...

Good luck. Remember all databases go to heaven!

5/25/08 1:03 AM  
Blogger lazzurs said...

Been there, done that. I am really sorry to hear you guys are not having a good time.

Everything is great when it is running fine but the real test of everyone is when it all hits the fan. Keep it cool and remember, it will all be over soon :D

Take care.

5/25/08 1:26 AM  
Anonymous Dan said...

I can only imagine the psychological hell you guys are going through. Stay healthy!

New user. Love the platform. Keep it up!

The commoners don't understand that rewriting a service complicated enough to found a company on doesn't happen overnight, and redistributing a database across multiple servers doesn't happen overnight, and buying more brilliant engineers doesn't magically solve the problems any faster.

You are doing the right thing by placating the community with transparency and giving it your best.

Thanks guys!

5/25/08 2:09 AM  
Blogger Mark said...

Good work guys, hope that you can solve the problem! keep up the hard work!

5/25/08 2:15 AM  
Anonymous tim said...

hey guys, its not a problem for dev people to fix that. The real Problem is the datastructure and that relational db that they use at the moment. It was made long time ago, for data in tables. But the structures today are things like a socialgraph. Smalle pieces of data, more querys, more connections and so one. But wait another year, help is coming. Why don't make a DB only for that? I know Germans they working since two years in this field. See http://sones.eu And twitter is not alone with this shit.. Talk to the cto's they know it..

Tim

5/25/08 2:52 AM  
OpenID potatohai said...

hope twitter will recover soon

5/25/08 3:29 AM  
Anonymous Greg said...

Wow! My first hours on Twitter and look at all the excitement & drama! I can really stay & watch whenever...? You guys really are cool, but some have yet to grow a sense of humor ;~)

5/25/08 3:35 AM  
Blogger Roi said...

Good luck guys!
twitter is being very big in the Catalan blogosphere. We really miss it when down.
Salut!

5/25/08 3:58 AM  
Blogger Kel Kelly said...

you guys rock the house! hope things work out soon. this is the best app in the web 2.0 world. i'm rootin' for you from the sideline.

5/25/08 4:10 AM  
Anonymous bootkidz said...

Good luck

5/25/08 6:04 AM  
Anonymous NoazDad said...

When you guys want to invest in a REAL database, give us a call here at Sybase. I mean, if ASE is capable of handling 90% of the financial trading traffic in the world today, I think it can handle Twitter...

5/25/08 6:08 AM  
Blogger Linden said...

Thanks for announcing the problem. I'd much rather get the news from the official Twitter blog than IsTwitterDown.com!

5/25/08 6:16 AM  
Blogger Bob Collins said...

I'm digging the notes on the page when there's a problem...with links to the blog post. A step in the right direction. Best of luck.

5/25/08 6:24 AM  
Blogger SG said...

one more reason i call 'em twits, not tweets. but hey, anything to attention-whore's fine w/me. LOL, you. go. twitter.

5/25/08 7:23 AM  
Anonymous Anonymous said...

Dang, db006 was the best of all.

5/25/08 7:43 AM  
Blogger Citizento said...

Hi, I like your service a lot and I hope you are going to recover soon. Now, forgive me for my n00bish question: why not moving your entire infrastructure to amazon s3? Thanks.

5/25/08 7:56 AM  
Blogger Frans said...

To all the armchair experts who know how to fix Twitter's problems. A wise man once said: "Better to keep your mouth shut an let people THINK you're a fool than to open it and let them KNOW it!"

5/25/08 8:11 AM  
Anonymous Anonymous said...

Good luck with things! Get up a donation system so we can help you guys out! I'm sure there's plenty out there who would donate.

5/25/08 8:59 AM  
Blogger businesssolo said...

Is okay thanks for good service

Keep the good work

Thanks
Mova Nugraha
Solo - Indonesia

5/25/08 9:10 AM  
Anonymous Anonymous said...

But are there any plans to scale up on servers? Twitter is a great idea and obviously very addictive, but it's not going to be worth much if y'all don't upgrade beyond capacity. Let us know what's going to happen in the future, please!

5/25/08 9:16 AM  
Blogger Vanessa said...

I'll be more careful from now on, promise!

5/25/08 9:31 AM  
Anonymous Anonymous said...

i get askeered when twitter gets lost.

5/25/08 9:49 AM  
Blogger Lou Mongello - DisneyWorldTrivia.com said...

"It's not my fault...." -- Han Solo

5/25/08 10:30 AM  
Blogger Trish | eMailOurMilitary said...

Just think what a great story this will make when you're on Letterman.

5/25/08 10:33 AM  
Anonymous Jeff C said...

Suggestion: do your fans a much needed favor by posting updates on your blog more than once a blue moon. Telling us Saturday morning that part of Twitter will be down for "a few hours" ... and now's it's Sunday afternoon with no additional updates, is abysmal communication. Posting on your blog several times a day during outages is not that difficult, and would go a long way to soothe the nerves of the many thousands who're rooting and waiting for this thing to actually work for more than ten minutes at a time.

5/25/08 10:38 AM  
Blogger Steve Garfield said...

Have you considered Oracle clustering failover technology so when one server goes down it's load balanced against the next server that's available in the cluster.

See:
RAC - Real Application Clustering


"Oracle RAC supports the deployment of a single database across a cluster of servers—providing unbeatable fault tolerance, performance and scalability"

"24/7 availability—Provide continuous uptime for database applications

On-demand scalability—Expand capacity by simply adding servers to your cluster"

5/25/08 10:53 AM  
Anonymous Anonymous said...

Come on, people. It's Sunday afternoon and I'm missing my Twitter-fix.

5/25/08 11:16 AM  
Anonymous BawldGuy Talking said...

Fellow twitterers -- they know exactly what must be done to rid themselves and us of the constant service interruptions. I like twitter a lot.

But reminding folks they're free is misleading, unless they're a non-profit organization.

Are they? I'm new to this and don't know. I feel badly for twitter, as this has to be drivin' them nuts too.

5/25/08 11:21 AM  
Blogger Ryo said...

This is getting funny... I don't think you're still in control of twitter.

5/25/08 11:36 AM  
Blogger Gleb Nikitin said...

As i know almost 80% of the requests come from API usage, as clients uptdate automatically.

Due to my calculations the current data that the clients could need is below 500 gigabites.

A simple solution would be actually to set-up a couple of MySql servers with load-ballancing and send all API-requests to them. The main database has to give updates only to one of the additional servers once a minute and they will just pass it on one by one. This is a simple solution but it will let you keep the website running untill you finish the redesign of your system be it segmentation or whatever you have chosen. Each server would give you at least 300 conects per/second assuming each querry is following 1000 people. It's the cheapest solution that can be implemented in no time and you can keep your RoR setup.

Oracle is surely faster, but it's always a question of price: would it be cheaper to invest in database and development or in hardware. At the current volume it's more reasonable to implement a cheap solution that would allow you to concentrate on the key development without caring of all the downfalls.

5/25/08 12:04 PM  
Blogger Nineta said...

Vaya!!! Que sepas, twitter que en castellano también se te echa un montón en falta!!! Recupérate muy prontito que nos perdemos!!!

Besos

5/25/08 12:18 PM  
Blogger Vonnie said...

I know as much about database management as I do about astrophysics (I.E. Nowt) but I do miss twitter when it's down.

Good luck fixing it guys! This is what happens when a great service gets successful quickly.

5/25/08 12:28 PM  
Blogger TIMSLIM said...

Twitter you are always worth the wait! ((((HUGS))))

5/25/08 12:36 PM  
Blogger Sameer said...

Guess yu guys need t wrk on a better disaster recovery plan... if yu havent had so ;) Have yu considered replicatins and RAC;s yet?

Regards
Sameer
P.S Twitter is the O in our lives... realise the missing o in the above text?

5/25/08 12:47 PM  
Blogger Kevin said...

Hey, why not blame the PG&E (that's Pacific Gas and Electric our local utility) outages for the crash? Everybody else points a finger (usually the middle one) at PG&E when something goes wrong.

;)

5/25/08 12:56 PM  
Blogger Admin said...

> "Does this have anything to do with RoR being unstable?"


Unlikely, unless Ruby or Rails can cause a database to actually crash.

Your question also presupposes something that hasn't been shown to be true. ("Facts not in evidence" or some such Law and Order speak... =)

5/25/08 1:13 PM  
Blogger vidsaw said...

Hi,

I'm sorry you are having problems, I still continue to support Twitter and will continue to do so during this down time and in the future.

Hang in there!

5/25/08 1:21 PM  
Blogger RokCK said...

Blur!

5/25/08 1:28 PM  
OpenID tof916 said...

damn yall!
mabey you should look into adult-web hosting! LOL j/k yall.

seriously, i agree with Joe Manna:

I strongly suggest twitter sets up geo-located Co-Lo's around the world and the states.

5/25/08 1:33 PM  
Anonymous victoria0918 said...

Wooops...unfair timing. Whose holiday vacation are we (i.e. freeloaders on Twitter) trampling on?

Thanks hardworking Twitter people. I will have a frozen margarita in your behalf =)

5/25/08 1:42 PM  
Anonymous Anonymous said...

http://jaiku.com

Twitter get your stuff together.

5/25/08 1:46 PM  
Anonymous Trnds said...

it is not good,that the "Older"-Button is out of oder :-(
I want my "Older"-Button back!!!

5/25/08 2:48 PM  
Anonymous Reed said...

Thanks for keeping us updated. To you whiners out there - yeah, it is very frustrating because we've all gotten so into Twitter...but it is FREE, Mr. and Mrs. Twitter are being honest with us, and if you think there's something better, go for it. If they had waited until it was perfect it wouldn't be there at all; scaleup is really difficult. The sun will come up tomorrow.

5/25/08 3:06 PM  
Blogger El Desaforado said...

D'ya think that this is the better way to ontain publicity?

5/25/08 3:46 PM  
Blogger chrispian said...

$15 Million just doesn't go as far as it used to ;)

I kid, I kid!

5/25/08 4:12 PM  
Anonymous Andina said...

A piece of advice: drop the fancy new offices and invest all that money on making twitter reliable.

Lately it has been a complete mess, being shut down once or twice a day...

guys: more work, less office space.

5/25/08 4:14 PM  
Blogger iellie said...

I just want to apologize for breaking twitter again Sorry...

5/25/08 4:55 PM  
Blogger Tim said...

It's a free service I enjoy for not paying anything.. So who am I to bitch if it's down once in a while..

I'll still poke fun at the elitist RoR bastards though, but not the Twitter folks.

You guys are ok. :)

5/25/08 4:57 PM  
Blogger Joshua G Woodward said...

This is both a blessing and a curse for twitter YEAH that you had so many connections but oops overload

looks like twitter will have to upgrade or something :)

5/25/08 5:10 PM  
Blogger Ketira said...

I noticed just *who* you are hosting with. Have you ever considered changing webhosts? I'm on Dreamhost (http://www.dreamhost.com/ ) and they have an excellent support team for things like this.

Just curious....

5/25/08 6:25 PM  
Blogger Erica said...

Thanks so much for the updates! I hope twitter starts to feel better soon.

5/25/08 6:48 PM  
Blogger Belle Chua said...

Bad forecast on the traffic control, but I still LOVE Twitter...Get well soon Twitter!!!

5/25/08 6:58 PM  
Anonymous leo1969 said...

If you don´t be able to put Twitter to work fine, so let it be and take your hands off. The web2.0 generation is to much for your brain!

5/25/08 7:09 PM  
Anonymous ajp said...

I'm always amazed at the sense of entitlement certain people have. Ignore the ungrateful bastards-- we love you, Twitter. Thanks for working so hard to give us something so fun. You're worth waiting for.

5/25/08 7:11 PM  
Blogger BCK said...

Any news? Its been a little more than a "few hours" and half of twitter is still dead. Also the IM bot was to be taken offline Friday, its Sunday when it is coming back.

And to all the people asking people to quit complaining: I've given it time, but I figured the kinks would have been worked out, as twitter has become more than a simple service, and having it down for an extended period puts a hamper on those of us for use it for business related dealings

5/25/08 7:25 PM  
Blogger Brent said...

Growing pains?

5/25/08 8:25 PM  
Anonymous Anonymous said...

1. Missing Twitter
2. These problems are going to make it easier to leave.
3. Don't care if free or 'paid' -- if it don't work...I'm outta here...
4. The masses are mooooooooooooving elsewhere.....

5/25/08 8:28 PM  
OpenID muertecaramelo said...

Dear Twitter: Get well, I feel sick when you go down... Im emphatic... and have a hell a lot of social issues.

Luvs from Mexico.

5/25/08 9:33 PM  
Blogger megalar said...

BCK said...

"Any news? Its been a little more than a "few hours" and half of twitter is still dead. Also the IM bot was to be taken offline Friday, its Sunday when it is coming back.

And to all the people asking people to quit complaining: I've given it time, but I figured the kinks would have been worked out, as twitter has become more than a simple service, and having it down for an extended period puts a hamper on those of us for use it for business related dealings"

WHAT A DOUCHE. How about you spend some money, get your self a simple hosting plan, install wordpress, and then prologue? I mean after all, twitter is free so you can't realistically complain and expect to be taken seriously. And since you use twitter for business surely you can spare $5.95 month for dreamhost and $7-10 a year for a domain. If not then your business model is obviously flawed.
Also, just what functionality is it that you need so badly? You can still tweet. You can still read tweets.

5/25/08 9:37 PM  
OpenID lunza said...

It's been longer than the "next few hours." We're starting to get into the next few days here. An update with an ETA might be nice.

5/25/08 10:25 PM  
Blogger Liquid_Skinn said...

i see.. :) well.. good luck! :)

wow, you better think about getting yourselves some new DB's. I invited all my friends to use Twitter..

Good luck guys! :) GO Twitter!!

ps. i really love your SMS support. ;) really helpful on boring situations. :)

5/25/08 10:31 PM  
Blogger Diane said...

Everything is going to be alright. alright...

twitter is great service.

once telephones were a luxury and now we cannot live without them.

same with twitter.

thank you twitter.

5/25/08 11:18 PM  
Blogger Nosfy said...

hurry up, twitter's daddies. i am seriously hooked.

5/25/08 11:32 PM  
Anonymous jenn said...

ahhh poor twitter! i'm not a fickle lover - good luck with fixing all the bugs! i'll be here for better or for worse. thanks for all the hard work!

5/25/08 11:41 PM  
Blogger CITIZENDANGERX said...

Twitter is cool.
Downtime is frustrating.
Calling someone a douche in this forum makes you an even bigger douche.
I would type some gigity-geek bullshit about servers or whatnot, but I wouldn't know my server from a hole in the ground.
I'm sleepy.
I like macaroni & cheese.

5/26/08 12:17 AM  
Blogger Nindya a.k.a. Kappa said...

Some functions still didn't work as usual due to the database recovery process but it's okay :) I love Twitter and I hope Twitter could get well soon :)

You guys at Twitter, good luck :D You guys rock :D Thank you :)

5/26/08 12:33 AM  
Anonymous Fiona said...

Thanks for your continued work on Twitter, most of us love you for it and can't wait until pro-teittering comes about <3

5/26/08 12:46 AM  
Blogger Wao said...

" Bless the server indeed "

@_@

5/26/08 1:01 AM  
Anonymous Anonymous said...

Why is everyone bitching? Twitter is a free service provided for your enjoyment - they go down and there's uproar? Who the hell cares, go outside and do something else!

5/26/08 1:45 AM  
Blogger Algernon said...

"Help" links didn't work before this problem, anyway...

5/26/08 2:13 AM  
Blogger funjon said...

From one horrendously busy systems architect/engineer to another, my condolences on the outage, and I hope you find the resources you need to get scalability and database problems fixed.

I've done the shoestring budget, and hopefully some of that $15M is earmarked. Unfortunately there appear to be a lot of people out here who think that money magically makes problems go away - oh, if only it were so.

So far I'm happy with Twitter's service. Keep up the good work.

5/26/08 2:22 AM  
Blogger Ilona said...

Sorry for that and uff, I thought it was me and I doing something wrong!
Gerhard Kaiser - Google me

5/26/08 3:24 AM  
Blogger Ross W. Lambert said...

1. Twitter's a free service. If you want a service level agreement, look elsewhere--and be prepared to pay big bucks.

2. Twitter folks... look into message-oriented middleware. I implemented load-balanced message queues at a telecomm and it worked fabulously well. It gives your system enormous elasticity.

Regardless, good luck. Twitter's a lot of fun and darn useful. Whooda thunk such a simple concept could hit critical mass?

== Ross ==

5/26/08 3:25 AM  
Blogger Reven said...

Bueno, espero que repongan pronto el servicio chicos.

5/26/08 5:40 AM  
Anonymous Anonymous said...

When can we except jabber to be back online?

5/26/08 5:55 AM  
Anonymous Anonymous said...

Anyone want a Jaiku invite? You can email me at 91AKMk3QZxCmZ1GB@spambox.us (Don't worry it's safe to email me there. I didn't want to post my real email address so I masked it with a free service from http://spambox.us )

Give Jaiku or Pownce a try while Twitter gets it's head out of it's arse.

5/26/08 6:08 AM  
Anonymous Pubdomains said...

Oh man... anything we can do to help Twitter !!!
It's been wonderful since the time I have joined and my best wishes to de-man.

Get Well Soon !!!

Cheers

5/26/08 6:22 AM  
Blogger Obsessed With Excellence said...

Awwww.... FREAK OUT! Come on, people, don't you think these guys are running around like mad trying to get everything working again? OF COURSE they are doing the best they can as fast as they can. The let all the IT guys out of the closet, and they put on their elf suits, and they are nibbling on the wires as we speak... go little IT elf, GO!

Get better, Twitter, we love you!

5/26/08 7:24 AM  
Anonymous Anonymous said...

I personally think you guys should remove IM posting capabilities. Obviously, you can't handle the number of connections.

Besides, with IM you can post at extremely higher rates than compared to that of texting or updating on the web.

5/26/08 7:29 AM  
Anonymous Anonymous said...

Twitter is good the problems may or may not be worked out

For those desperate for Twitter to be 100% up well Get a Life if your day revolves around Twitter then better relook at your priorities.

Alt to Twitter - Pownce, Friendfeed
others Twitter is not the Be all end all

5/26/08 7:34 AM  
Blogger Ken said...

Clearly the sustaining business model the gives investors the incentive to fund Twitter hasn't carried through to the long tail of service deliver just yet. Those of us who've been using Twitter since the very beginning (pre-SXSW) wonder when it ever will.

Twitter's great and we all love it, but at some point it's either going to have to find a reliable model, get some leadership that's open and honest about how long problems take to resolve, figure out what it's TOS really is, or it's going to start really suffering.

5/26/08 7:48 AM  
OpenID bpbrainiak said...

good luck boys I hope you fix it soon

5/26/08 8:01 AM  
Blogger Adictísima said...

Any new info about this? I think this damage is more important than a few hours.

Best regards,

5/26/08 8:15 AM  
Blogger Tim Harper said...

@The Mad Doctor

Dude - he just said the DB crashed from too many connections. Last time I checked, "Ruby on Rails" wasn't a database server :) Withdraw the bigoted hostility and leave your python long enough to take the red pill and discover why so many frameworks are copying Ruby on Rails. :)

5/26/08 8:29 AM  

Links to this post:

Create a Link

<< Home