Too much Jabber!
We found an errant API project eating way too much of our Jabber (a flavor of instant messenger) resources. This activity (which we've corrected) had an affect of overloading our main database, resulting in the error pages and slowness most people are now encountering.
We're bringing services back online now. Some will be slower than others for a while, and we'll be watching IM and IM-based API clients very closely. We'll also be taking steps to avoid this behavior in the future.
Thanks for your patience!
Update: We're turning off IM services for the evening (Friday) to allow for the system to recover. We hope to turn things back on Saturday.
84 Comments:
Thank god for that :)
I have been waiting to tweet for an hour! :D
Glad you found the problem. :-D Or at least one of them?
Despite all the flak and bitching, we do appreciate all the hard work guys. Thanks! :-)
Definitely still not working ... it's been out all afternoon. It's nearly 7 where I am :(
I hate hating on twitter, but engineering wise this is a disaster. I've been with you guys from the beginning and the lack of resiliency is dismal... not only fixing bugs but staying afloat (laughable). My patience has run out and i'm dropping your service. You toyed with my loyalty while others reward you with gifts and adulations that i think amount to "it's okay your screwing up".
good bye and good luck. waiting for twitter to join the dead pool!
I kind of have to agree with anonymous. It's time to move beyond Ruby on Rails. It's not working. The hacks you've had to use to remotely scale have only brought new and more annoying problems.
How much longer do we have to put up with this. From a usability standpoint (when it actually works), Twitter is far beyond the competition. But if its impossible to use the site, does it even matter?
Hey Jack,
If Jabber programs keep overloading twitter, why not block these programs until a better way can be developed to make it more stable?
Just a thought...
Jack, the transparency in how you're dealing with Twitter reliability problems is greatly appreciated. Despite the anonymous comment above there are thousands of people wishing you the best of luck and willing to stick with you through the scaling trials and tribulations.
-Brian
Twitter is the kinda beast you just want to see someone put a buckshot to and put it out of it's misery.
At the 2nd anonymous poster:
More hating on RoR when it's not Rails' fault. The backend messaging system for Twitter has nothing to do with Rails. Rails is only the frontend and scales just fine for that particular task. It's the backend bits that are failing so often and are resulting in this slowness. They have more systems to build in order to support this kind of infrastructure, but it's definitely nothing you can pin on their usage of Rails.
And this is coming from a PHP guy!
Can you confirm that this Jabber client wasn't Twitter4R in disguise?
Because I discovered some badly drawn code this afternoon in my application.
Thanks,
Anonymous Coward using Twitter4R
Thank you for the explanation! Your humor on the outage messages was appreciated. My time on FriendFeed this week has brought to light just when and where I love your service and how far everyone still has to go in this space to mash the right stuff at the right time with the right security.
Hope you get to enjoy more than pizza this weekend! I really am grateful for your time!
Thanks for the heads up and the hard work, boys and girls. Sure, there are outages and error messages, but I'm sure I'm not the only one who firmly believes you'll eventually prevail in your quest for long-term stability.
It amazes me when people moan in response to these type of informational posts. 'toying with your loyalty' come on, you are kidding right?
I'm sure the vast majority, (not likely to post here) are grateful for the service and the work that goes into it. Not to mention the transparency and updates about outages that are provided on this blog.
I really like this free service and appreciate the work that goes into providing it. keep up the good work.
Damnit; for all the money I pay for twitter, I demand better service!!!
Wait. nevermind. It's free! Guys; you rock. Do what you can when you can; it's a great system
seriously guys, the updates about what's going on in twitterville are giving me a renewed faith. Sometimes all we need is to know how things are going, rather than logging on to an error message
Continue the transparency! We all appreciate and love you for it! :)
Thank you for the increase in status/issue updates. I know the overwhelming majority of twitter users appreciate them.
It seems like you have become more transparent this week.
Your customer retention numbers will increase as a result.
This post has been removed by the author.
I have to admit, these issues are becoming unbearable. I am not going to hide my name, because I feel that this is a serious issue. I use the IM service religiously because I do not want to have to visit the site constantly to get my updates. And I also don't want to use a third party program, because I am not that dependent on Twitter, no matter how much I like it.
Twitter's downtime is massive, and is topping charts of social websites' downtime lists. Truly, I would hope that Twitter would work harder to prevent that at all costs, and so far, through the numerous improvements that have been made so far, progress on these issues still seem to be going nowhere quickly.
While I will continue to use your service, for now, I just wanted to voice my displeasure at the moment. While I respect your staff for the job that they do, I am still disappointed at the errors and issues that have befallen Twitter.
I need to start following this stuff more closely because I use IM as my main way of tracking Replies/Direct Messages, and wasn't understanding why I was getting no replies. Oh well off to use the Web.
Why don't you guys setup a durable buffer, between the Jabber service and the Database?
For example, you could employ something like Java Messaging Services (A simple Queue) As messages come in from the Jabber API, stuff it into the Java Message Queue, you can simply have a 1 or 2 message subscribers that pull the messages out as they are populated.
This gives you a way to buffer out spike traffic from the Jabber API and queue messages durably until they can be inserted into the Twitter DB.
RoR is great, but it doesn't solve ALL problems. I do a lot of work with rails on my front end, but why re-invent the wheel for things like messaging transactions, etc. when there are solid and open solutions out there.
Think of a messaging queue as a buffer between your Jabber API and the database. Like a bouncer, only letting 1 person in at a time into the popular bar. :)
Keep it up. We all know you guys have had your share of bad press. Regardless I love your service after only a couple of weeks of use. There is so much potential here!
That does it! I demand you refund every penny I've paid you for this!
Oh, wait, this is free, isn't it.
Never mind. Carry on.
(Seriously though - the ability to send and receive through my jabber server was the major attraction that got me to sign up. Hopefully the problems will be solved soon?)
such is life. hopefully you can get over this.
yeah it sucks when stuff breaks down and goes freaky, but how do you know how to fix the problem if you have never seen it?
im pretty new to twitter so this may have happened before. but i figure its just growing pains. and as for those who dont like twitter, dont use it, very simple. instead of complaining...
Thanks for clearing things up. Much appreciated!
affect -> effect
Thanks for the update. It is really nice to have answers to downtime of Twitter. As you are probably seeing it is not the outage but the communication of the outage that matters.
Thanks for the great service!
Billnad on Twitter
I missed my IM tweeting! Thnx for fixing it Twitter
All I gotta say to the haters out there is: how much you paying to use Twitter...? That's what I thought......
It'll be to see "What are you doing?" instead of "Something went technically wrong"
Those anonymous comments above are ridiculous. This is a free service for heaven's sake.....get over it. Work with it.
You...who ever you are...are idiots. Put me on record for that. Hate me too. Hate the services I put online. Who cares. I HATE whinners. Especially the ones who don't have anything positive to add.
My message to them:
Good bye. Won't miss you at all.
thanks, good job!!!
I for one appreciate the FREE service that is Twitter. A few days of hassles is easily forgiven. And thanks for letting us know. Hope you have a nice long weekend.
Twitter is the Robert Downey Jr. of internet apps
All these anonymous commenters hating on Twitter is cowardly. If you don't have the balls to own up to what your saying then don't write anything. Good riddence.
Seth G.
Ignore the Anonymous.
Probably aren't even users!
Twitter is awesome in so many ways. It's definitely worth the wait. I personally appreciate it. Keep pushing.
Seriously guys, I think it's about time you step back and take some advice from the most wise of posters. These people have debugged your problems from afar without access to the system. That's no small feat. And if that wasn't enough, rather than wanting to take credit by smearing their name all over the post they swallow their pride and leave it off. It's time for a little humble pie and who best to serve it than the most humble of us all. You know who I'm talking about. Yep, the anonymous.
Oh, and what the hell are you guys doing with my money anyway? I'm not paying you for nothing. There's too much white in your eyes. Type damn it type!
I just came back to use twitter after some time and noticed it was very slow. glad you guys did something about it :)
I wish East Coast Vegas would broadcast again. He hasn't done a show since the lame attempt of one in that New Orleans hotel room.
Sorry, but having Jabber/IM ist essential for me. If you turn this of, or limit it, you can turn off whole twitter.
I think the only reason to do this, is a threat to liberate twitter via a mashup of Jabber and RSS, like mentioned by Michael Arrignton on Techcrunch: http://www.techcrunch.com/2008/05/05/twitter-can-be-liberated-heres-how/
Isn't it strange (and typical!) that all the "anonymous" comments were from people with complaints. If you're going to bitch about it, at least be adult enough to leave a name. Sheesh.
Wishing you all the best in getting it sorted out.
First : Thank you for Great Service.
Second : Hope IM Service comeback Online Soon.
it would be nice if you guys would answer API whitelist and user agent requests... ;)
Can't you set a burst limit to tweets coming from IM's or so?
Keep up the awesome work, you guys (and ladies) rock.
to 1st anonymous: why the whining? it's a free service, your loyalty wasn't asked for or expected, so why make a fuss about your going away? can't you do it quietly?
ok, i'm doing the same kind of whining about your comment now :)
err... I really hope that wasn't me :(
errr.. I really hope that wasn't me :(
Thx
Seesmic's twhirl is working on a xmpp twitter integration. Could it be because of that ?
I have to agree with Brian Caldwell above, I appreciate the transparency and the straight forwardness. Here's hoping this doesn't become just a bunch of "RoR is teh suX0r" commentary. (Notice how its all the anonymous cowards that are bitching.)
thanks timdorr... the only thing worse than angry nerd rage is misplaced, inaccurate angry nerd rage hahah
Twitter, keep up your good work.
Most start ups are like these cute kids that everyone adore, but once they achieve critical success, they are like pampered, brash and irresponsible teens. Some of the most successful companies are incidentally the most hated ones.Most companies falter because their only objective is to make more, more and more money, so the focus shifts from building great products to commercially exploiting it. Did'nt granny say greed is bad. The only companies that survive the test of time are the ones that stay focussed on building great products. The good thing about Twitter is it simple but elegant UI and overall user experience, please don't mess with the UI. Also, the lack of ads is another positive.
I wish Twitter greater success and monetary returns. But simply hope that it doesn't spoil the great user experience by aggressively commercialising it.
Regards,
Twitterer
@snowbear [second comment]
I agree. thanks for the explanation and despite all our bitching. we still love you and respect you :)
you've heard this a bunch, but forget those jerk-ish comments about "i'm LEAVING this free service cuz you guys suck!".
I have no clue as to the complexities involved in running something like this, and I suspect those people dont either. you guys DO know, and are working on it. just keep going. Twitter is going to be a model of this type of transactional system one day!
Thank you for the update :)
Have a nice day :)
Hope this takes care of the twitter db problems for a while, been hard to get a message through off and on for a week or two now.
LOL. Let's all just Tweet in here. Maybe some more anonymous twits will come back and entertain us!
Hey Twitter, it's annoying that your service goes down on me more then what my partner does. It would be a full time job for her to meet your standards.
However as Twitter is a free service, I and the rest of us really can't complain. When Twitter is usable, it's one of the best services on the Internet to keep in touch and know what's happening (live).
As a developer myself, I've been keeping an eye on the challenges you've met and how you've over come these challenges. No one else will be successful to achieve a system like yours without failing continuously either.
So cheers from the community for what you've created, although I will look forward to Twitter working 100% 24/7 but I'm not complaining.
I find comments, like those of anonymous, to be hilarious. First off, this is a free service. I know that isn't an excuse, but it is free. The amount of traffic you guys take and generate is astounding. As a web application developer, I know first hand how tricky of a balance this can be. Granted, if I were paying for the service, I would probably expect a bit more, but not necessarily. All online applications have growing pains, especially those that allow for so many different ways of interaction.
Secondly, after seeing the post about looking for more network engineers, I believe you guys are taking the right steps to improve and grow the service. All the work you guys put in is greatly appreciated, and like any service, you only usually hear the complaints when things go wrong, not the praise when things just work.
Good luck to you guys, you don't deserve that crap dicks like Arrington write about you. Go Twitter!
Also, thanks for Ruby on Rails support. We both know RoR is not the source of Twitter problems, but heh, "they" don't stop.
as I said in my Twitter: OK twitter shut off IMs "for the night" its morning now still no IM?
Are you kidding? It sounds like some of these idiots are depending on Twitter for their livlihood instead of just a way to have fun. Lighten up and give them some slack!
Can't you add more servers to help the issue?
Effect, not Affect.
Sorry to be that asshole, but it was bugging me to death.
Good thing we don't pay for this service. I understand that Twitter is a victim of it's own success, but these scaling problems are just dragging on.
I must admit, I use the instant message feature the most, so I could see where it would be a busier port/api.
I have faith that you guys will get it back up. Every site has growing pains, and I'm sure this is just yours. I think many people don't realize the hard work it takes to keep a website (or related back-end services) up and running, and thereby expect that it should be available 100% of the time. Downtime happens, it's the curse of any System Administrator :).
If you don't like this site, leave. Personally I use this to keep up with friends and it's for fun.
I can't understand all these outraged people. Twitter is not a 911 service, it's not made for saving lives (yes I did hear about the guy who was arrested, but that was one case), it's to have fun.
If there is another awesome appp like this for free out on the web, I suggest you hop over there and use it.
Oh yes, and it's FREE.
Hmm..no wonder i keep seeing birds tweeting all time. Lucky that you are fixing it :)
Guess i have to wait for a little more while.
Hey folks, I just hit the wrong button in Blogger and deleted six new comments to this post. Luckily, I still had them in my email so here's what they said:
Matt said:
"I like Twitter, I have friends that like Twitter. The question is, do we like it enough to stick around when it doesn't work for long periods of time? @Jenny, perhaps you misinterpreted disappointment as outrage. Don't toss too many accusations around yet. I'm not angry, just disappointed that I can't post through Digsby right now, my main posting method. Regarding the free part, would you pay for Twitter in its current state? If I were paying for it, I would be tempted to cancel my subscription. I think Twitter is free because it has to be, so that argument really holds no water. Anyway, that and 2 bucks will get you a coffee at Starbucks."
Kai said:
"Id love to see the anonymous haters build something better, support it and pay for it themselves so that it has 100% uptime and remains completely free for all of us. But then, I doubt they have the know-how of the systems required to build such a robust backend that doesnt fail, let alone understand the operational costs of enterprise class resiliency.
They do love though to get on their little soapbox (as if they have something to stand on to bitch about) they are the type who think they shouldnt have to pay for these kinds of services and still be able to expect and demand perfect uptime records. Grow the **** up haters, it costs money to run and time to support these things, if youre not willing to contribute either then you really dont have room to talk now do you?"
matt said:
"I have no cause to complain, it's a free service. I'm not mad that Twitter is down...I'm just getting the shakes. I think that is why everyone is bitching; it's such a great idea that it has become electronic crack. When it gets taken away, the users get crabby.
I'll wait patiently whilst the back end problems are worked out. And I will probably tremble and get cold sweats while I'm waiting. Dang, I'm addicted. Come back, Twitter!"
kara said:
"I miss my twitter -- I'm so
addicted! Hope you can get things up and running again soon. Thanks!"
Omi Azad said:
"Still it's not live. Not only the IM the whole service are down now. :("
robreed said:
"I get really annoyed by the argument that "because it's free so users should be expected to put up with just about anything". Free isn't always free and twitter is a perfect example of that. There are a lot of people who have made considerable investments of their time, efforts, workflow, etc in the service. For people who have invested hundreds of hours, free is not an apt description.
Also, the community clearly represents very real value, as does the cachet afforded to those associated with the service due to its popularity. Where users are responsible for driving the success of services like twitter, I really wish people would stop dismissing the contribution. I understand I won't be inline for any $ should someone come along with a check for the principals but a little respect is in order. And saying FU its free if you don't like it just leave is not respectful.
When the IM traffic was blocked the service responded very favorably, no? Can't these different sources of traffic be mediated?"
Oh my god, you guys are terrible. Twitter has been online for for like 2 years and you still can't keep it online for more than a day? Someone is actually using your service and that causes it to crash? What the hell have you been doing.
So many different perspectives, different attitudes - this one comment thread alone is enough to write a psychology book on...
Simple fact of it is, Twitter costs us nothing... except, sometimes, entire days Tweeting away because it is so very much like crack.
To me, Twitter is a business tool. I use it to network and find clients. In fact, all of my clients who were not referred came through Twitter.
You would think such a use would make me royally ticked that Twitter is down. Nope. Never put all your eggs in one basket.
...besides, you've given me enough clients that I almost didn't notice the outage because I've been so busy making money...
Good luck with the Twissues. And if you guys ever come out with an "Enterprise Edition" with features like network-wide keyword notifications, interest aggregation, etc - I'm definitely game. Even if it's $20+ a month.
Twitter is absolutely useless without 99.99% or better uptime.
Hi, yep i'm the first an "Anonymous said..." at the top of the page.
It is free, I get that. I appreciate that. I'm an engineer too. I build software. I'm a professional, I rely on users to provide income etc... so in turn I should at least have the gall to provide a decently reliable service if they are providing for my livelyhood.
It's not only the uptime that's the issue, tranparency aside... it' how they address issues and the speed with which they do it.
May people including myself rely on this service, uptime is not to much to ask. For those who don't know, saying "hope you get better soon" are setting your limits to low... and this shouldn't be the standard with which all software developers think is acceptable. ITS NOT!
Start using DISQUS for your blog and discuss anywhere.
http://www.disqus.com
Nice to be humble and telling the truth.
Mr.G prods the twitter thing with a stick while quietly humming,
"don't it always seem to go
that you don't know what you got till it's gone
pave paradise
put up a parking lot"
and grooves out on the Oooooh la la la la's
Just as well. this was way too long for a tweet anyway. Here's sending you all some deep psychic debug-insight Concentration and Epiphany Karma!
Still not back up?
Is IM on Gtalk back?
http://bp3.blogger.com/_E8ZD85Wzu9E/SDMyvhFoPdI/AAAAAAAAAR0/8EwjfrS27-U/s1600-h/twitterdowntime.png
Still can't post to Twitter from Gtalk. But thanks to this blog, I now know that the outage is worldwide.
Is it back up already? :-/
I miss twitter working in all its glory... :(
In regard to my previous comment, I refer to the "twittersync" application from Facebook, not being able replicate our twits... Perhaps that particular API project could be switched back on?! *pleeeeeeeeease*?
http://www.facebook.com/apps/application.php?id=6009973148
Links to this post:
Create a Link
<< Home