We had a rough night. But now we're back.
I just left the office, at 6am, with most of our engineering and technical operations team. That's only a handful of people, but we were all there all night.
You may have noticed we had an outage last night/stretching into this morning. This was a planned maintenance project, not a surprise happening for us. But we did go far beyond our planned time window, which sucks. What sucks more is the series of outages (planned and not) we've had lately. We know this makes Twitter frustrating to use, and we want you to know, we hear (and share) your frustration and are working really, really hard to fix it.
The good news is we finished a major infrastructure project tonight, which we've been working on for months and that we think is going to help a lot. I'm sure there will be some glitches in the coming weeks, but we've laid some important foundation that's going to help us build toward the future.
Thank you for your support and patience.
Update (8:30a PT): We're still experiencing patches of slowness service-wide. Some of our systems are not cleanly restarting. We're watching closely, bringing things back up manually, and fixing our automation tools accordingly!
You may have noticed we had an outage last night/stretching into this morning. This was a planned maintenance project, not a surprise happening for us. But we did go far beyond our planned time window, which sucks. What sucks more is the series of outages (planned and not) we've had lately. We know this makes Twitter frustrating to use, and we want you to know, we hear (and share) your frustration and are working really, really hard to fix it.
The good news is we finished a major infrastructure project tonight, which we've been working on for months and that we think is going to help a lot. I'm sure there will be some glitches in the coming weeks, but we've laid some important foundation that's going to help us build toward the future.
Thank you for your support and patience.
Update (8:30a PT): We're still experiencing patches of slowness service-wide. Some of our systems are not cleanly restarting. We're watching closely, bringing things back up manually, and fixing our automation tools accordingly!
posted by Ev. at 6:07 AM
59 Comments:
keep your heads up, go home, and get some sleep :)
Hi, thanks for the peek behind the curtain. I think sometimes users forget that there are humans behind Twitter, and that right now, no one has to pay for the service.
I certainly never thought I'd use it and like it as much as I do, and thank you for all of your hard work to keep it up and running for all of us with the addiction. :)
You are the operators of our pocket calculators.
Well Done and Congrats to all the team.
Hopefully the issues will now subside and we can all enjoy Twitter.
@patphelan
Did this have anything to do with Joyent? You just praised them, then had a rough night. Kind of like when you say...Thank God that didn't happen, and sure enough!
Thanks for the info and all the hard work! I have grown to fondly appreciate the value of Twitter, and I excitedly share this site with clients that are looking for ways to communicate VIA a microblog. Keep up the great work. I remain another Twitterholic!
Jeff Tippett
www.calvertcreative.com
Well yeah, it took longer than it was supposed to take, but I guess it should be okay. You don't know how much people get done when Twitter ist down ;-)
It has been frustrating, and for us it was during the day, but we still love Twitter!
you had a planned outage during the Republican debates? Perhaps you need to better schedule your outages...
glad that it is up again. high server load crashed the server?
I think you're still down. I haven't been able to log on since yesterday afternoon.
I wouldn't say back quite yet.
We understand there are issues, but posts like this don't actually make us feel better. Transparency into the problems would... What was the issue? What was the infrastructure project? When's the next planned outage? How about an "uptime meter" on a status.twitter.com page? "We've laid some important foundation" sounds like a lot of handwaving when you've got a techie audience. People want to know what you actually did. I think that would go a long way.
Twitter's a great service, a few growing pains are only to be expected.
Thanks for making what is surely one of my favorite websites!
This is all well and good, and yet here it is 11AM eastern time and Twitter appears to be down due to "technical difficulty" again :(
Keep working on it! We are rooting for you.
Hope you can get beyond these glitches before folks move on. And hope things can get on track before SXSW.
I don't want to have to reinvest time in another twitterlike service.
doesn't look fixed to me. been trying for 12 hours to get set up. as a first user, i'm not too encouraged to stick around.
It's either still down or it went down again, as of 10:00 am central time.
yay! thanks team twitter for all your hard work and dedication. twitterers like myself really appreciate everything you're doing.
only when there are outages and i begin to suffer from twitter withdrawal do i realize how much twitter has become an integral part of my life.
ganbatte! ^^
Looks like you're still having issues - as it's 11 a.m. MT and I'm still getting the 'something's wrong' banner.
:(
Sorry such a rough night guys - nothing like a change outage that gets out of control... My sympathies!
Hoping you are back up soon!
We miss Twitter!
Twitter f'n sucks now. You guys pick the worst time do service upgrades. Even after whatever you did, it's worse than ever.
Thanks guys - you're not back yet though... I can't login. Keep up the good work!
I have to agree with the more negative commenters! You're in the big leagues now and your choice or maintenance window and your inability to get your work done within that window does not reflect well on your team. I am more surprised when twitter works these days (esp. via facebook) than when it doesn't.
"....don't it always seem to go, that you don't know what you've got 'til its gone..."
You can count us among the many who don't fully appreciate Twitter until our e-mailbox fills from those suffering LAFD Twitter withdrawl.
Kudos to those who gave up a night's sleep - and a likely a lot more restless nights prior, to make the magic better.
Respectfully Yours in Safety and Service,
Brian Humphrey
Firefighter/Specialist
Public Service Officer
Los Angeles Fire Department
LAFD Blog: LAFD.ORG/BLOG
1:40PM eastern and yeahhhhh still not doing so well. Somewhat vexing.
Down again! (12:45 central)
Right now it looks like you are about to enter the 15th hour of issues, I would suspect not the best for all concerned.
I love you guys but something needs to be done about the stability of the maintenance windows.
Also a bit more communication to users would be fantastic to help us understand your pain.
We rant but we still love you!
its not working ;-)
Dear Ev,
Frankly I’m baffled. As I type this, Twitter is throwing a 503 error back to twitterific and is showing the newly idiotic "Something is technically wrong" web page at the home URL. This is the morning after yet another period of system maintenance or upgrades.
Your goodwill is rapidly deteriorating. The service is notoriously unstable, and in fact just yesterday I saw someone begin advocating against depending upon Twitter for emergency communication. Why? Too undependable. This is quite a turn-around from the period during & after the San Diego fires, where Twitter was embraced by fire dept folks for use in just this type of situation.
It would be easy to say "You get what you pay for" - neither I, nor any user, pay to use Twitter. And you’d be right. But you’re trying to make this a business, and so I respectfully suggest that you begin running the operation as though *everyone* is paying. Bring in some experts. Redesign - from the ground up if necessary.
Once folks develop a habit of use, they begin to feel entitled. They show up, you provide a workable twitter service, and somewhere along the way you figure out how to monetize. That’s the value exchange, and is essentially what Fred has publicly stated abut his investment. But when they [users] show up and all they encounter is a timeout (there’s the 503 again just now in twitterific) or a condescending error page, they feel cheated. People are funny that way.
And they will leave. As I’ve said before, I really like Twitter. But the value exchange is currently askew.
(oh, and there’s that 503 again… three times while I was posting. )
John Minnihan
Maybe Twitter ought to consider dumping Joyent. I left them for another host because they weren't even dependable enough for my blog. Daily outages are unacceptable, whether you're running a blog or a messaging service that has become essential for a lot of people.
Those guys who give you a hard time are foolish. They're getting this service for free, and yet they feel like they're owed something. Who cares if Twitter goes down for a day or two here and there? Get a life, go outside, enjoy the world. It's just a website on the Internet. No one's going to ask you when you're 60 about how many tweets you wrote. And if you're depending on it for business, then you really need to take a hard look at how you run your business. Perhaps relying on a social networking site isn't the best choice.
Hey everyone!
Downtimes happen and guess what - if you can't tweet right now and that makes you upset, imagine how it makes Ev and the rest of the folks at twitter feel? A lot worse then you.
And quite frankly relying on something like twitter for emergency services seems a bit out of scope. No offense twitter, but I'm sure you weren't thinking of replacing 911 either.
Keep up the good work twitter team. I feel your pain and look forward to future tweets.
@charlie - I agree. A look under the nest would be fantastic. Let's see some transparency for the folks who care.
Everyone knows you guys are busting your humps to get this service rock-solid and understand the stakes. But I'll say what I do with my video-game expert 13 year old son when he doesn't achieve, follow-through or focus on the #1, most important thing in his life right now (school): it's the OUTCOME that matters.
"Buddy, as a kid it's great that you tried hard. In the real world it doesn't matter that you worked all night, didn't play video games because of homework, or that you deserved an A simply because you worked so hard. What matters is the outcome of that effort."
He loves Halo 3 but if he achieved a level and it was bug ridden, he'd say "this sucks," wouldn't continue and he'd never buy another Halo game. My iPhone is cool, but if it crashed and didn't work, I'd choose something else and the product would die and deservedly so. If my relied upon car was awesome but the engine stopped two or three times per week for no apparent reason, I'd ditch the damn thing. Same with Twitter: I feel for you and your hump-busting, but the outcome we users expect from ANY service is that it works and is reliable.
Of course, the problem with most 'free' Web or 'net-centric communication apps is that the mantra is, "Let's get to a critical mass of users and then monetize it" but scaling and reliability takes money. If I was paying for a service, I'd be demanding but instead am going to see what unfolds with Twitter before doing much more.
Ever heard of a rollback strategy when things go wrong? Blame things on technologies if you want, but Twitter suffers from bad project planning. They have a planned upgrade with no fallback strategy. So if things go wrong then let the users wait. And wait, and wait...
Some more insight into your infrastructure would be interesting, as some people said. But I don't think you're obligated to do so, that's silly talk.
inmates,
Rollback is neat, but it's a fairly expensive and complex thing to do. Especially for a small startup.
i am still not receiving updates in my phone here in brazil. my network is TIM.
A whole day without twitter is frustrating.
Hoping to see it run full steam tomorrow with IM too. and please enable the notifications automatically - i dont want to unset n rest my gmail account n cell too.
Twitter is great, and people should stop whining about some hiccups. Being temporarily disappointed by downtime is understandable, but folks should think about what it would mean if Twitter did not exist.
This is the transition between proof-of-concept and the big leagues. Every single company that goes makes his shift can see problems just like this.
These issues WILL be fixed. Thanks for the effort, Twitterfolk. Keep banging on the machinery. We'll be here when you're done.
Did you run a level seven diagnostic?
Looks like you guys finally did something right and left those incompetent clowns at Joyent. They couldn't web-serve their way out of a paper box to save their lives.
I totally love twitter - in that it is an entirely new form level of communication. I have it on almost all day and as my SO can attest late into most eves.
The errors are a little frustrating - but really not a big deal... I think there are a lot of bitches out there who should get out from behind their computer more often - and say thanks to Ev for everything thats provided to them for free while they give so little back other than complaints.
scaling is hard but not as hard as having a great product...love the service
John Furrier
http://furrier.org
twitter: furrier
This is a bit curious, given that (a) the post yesterday was about how great Joyent is, and (b) the post today over at Joyeur suggests the major infrastructure project you've been working on for months is, uh, migrating off Joyent.
You'll understand that these two things come across as somewhat contradictory. :)
I appreciate that your staff put in a long night, but I'm concerned about some items, especially after reading Dave Winer's post and Biz Stone's reply.
OK, perhaps you shouldn't be completely transparent with EVERYONE, but do you have, or have you considered, setting up an advisory panel of users under non-disclosure to provide feedback? Not only on technical issues, but also on issues such as monetization that you are currently deferring.
There are dangers in a company (especially a small company) going it alone without advice from outside, and while there are also dangers from completely opening the kimono, there is GREAT value in convening a small group of people under non-disclosure, and saying, "OK, here's how we plan to address the performance problem, and here's plan B if plan A doesn't work out. What do you think?"
More of my thoughts here.
My recommendation: fix the problems yourselves, avoid entanglements with other vendors, maintain independence, and continue being up front about your issues. The risk is attrition of users, but I think what you're doing is worth losing a few self styled heavy hitters. I like what you're doing; so do a lot of other people.
Dennis McDonald
Alexandria, Virginia USA
http://www.ddmcd.com
We value twitter very much in my circle of friends and with my colleagues. We feel the withdrawal pains when twitter is not available, but are in your debt for the service.
I am just really glad it was up and running during the State of the Union.
Thanks for the hard work guys, you're only human. I won't deny frustration at the outages, but I completely understand...you're a startup with a lot of traffic (from all of us who love you) and sometimes stuff happens. I also understand that it's not always in your best interests to print out your technical specs and get approval from obsessive Twitterers who request it. You're a company, you'll do what you think is right.
I do think a tad more communication would make people happier. Give more deets about the situation and throw some humanity in! The touch about leaving at 6am really helps us understand that you are human beings and doing your best.
Now get your human butts to bed and get some sleep. :)
Just a side note.
I still stand behind my last post, but all the mentioning of 'just a startup' got me thinking.
No offense, but I think twitter past the point of a startup awhile ago. Anyone look at their stats lately? and if its true that ALOT (70% comes to mind - don't quote me on that) of traffic is coming via the API then sites like compete.com wouldn't even show the half of it.
And Obvious (the creators of Twitter) is the company which founded Blogger... they have seen traffic before.
Something went wrong somewhere. And I highly doubt it was a traffic jump. New systems are never fun.
Welcome to Verio!
I try not to speak to badly about any company trying to keep servers running, but when TextDrive was absorbed by Joyent, it wasn't long before we had to pack up our personal belongings and go elsewhere.
I spell out our history with TextDrive/Joyent and our long term love affair with Verio here: http://www.barefeetstudios.com/2008/01/31/twitter-joins-us-on-verio/. I'm posting the link because the trackback links at the bottom of your post are all broken. I hope an upgrade to WordPress is in your future as well. (I know Ev created Blogger. But it sure is broke a lot.)
you probably don't need another post saying 1)we love you 2)this is unacceptable and 3) get some sleep, but here it is. Suggest you take Winer up on his offer to help. Twitter has changed my life, and I'd gladly pay to have it operate reliably.
there's still a problem with gtalk users, isn't it?
has that been informed and will it be solved?
Figures, the night that I'm in the hospital with my wife having a baby, I wanted to use twitter to update the family, and it's down.
Oh well, off to Jaiku for tonight.
Take your time and drive safe when you do go home. Fatigue can be as deadly as drunk driving.
The negative commenters can suck it anyway.
Well seems like you guys are back now... it was hard to live with you but I realize it was necessary upgrades.
I'm sure very few Twitter users understand the whole infrastructure thing anyway. Something like Twitter hasn't been done before...I remind myself that when I can't get through, because it's all about pushing the limits of the technology.
Kudos to you.
The IM bot on gtalk and phone notifications still not working.
I hope someone is looking into this thing!
twitter seems to be running slow again. Taking hours for new tweets to show up in third-party apps (twitteriffic, facebook, etc.). Is something being worked on?
Hi, thanks for the peek behind the curtain. I think sometimes users forget that there are humans behind Twitter, and that right now, no one has to pay for the service.
It has been frustrating, and for us it was during the day, but we still love Twitter!
Post a Comment
Links to this post:
Create a Link
<< Home