At half past midnight Eastern time on December 12th, our site started generating server side errors that prevented users from being able to use our service. Unfortunately the outage wasn't a full outage. The web site was operational, but would serve up error messages instead of fulfilling user requests. As a result, our monitoring system, that, at the time, simply tested to see if it got a response from the web server, did not send a page to our network staff to let them know that there was an issue at hand that needed to be taken care of immediately. This incident only got resolved when our staff came to the office in the morning and noticed the issue. It was quickly fixed after that, but left tens of thousands of you stranded in the interim.
That's not the experience that we would want you to have when using SceneCaster and we are taking concrete steps to make sure that this scenario can not occur again in the future. More specifically, we have now set up monitoring that checks the content that the web service returns rather than simply making sure that it returns something.
We understand that many of you were frustrated, especially the new users that had just signed up for the service, and would like to apologize to all of you who experienced difficulties using SceneCaster as a result of this outage. We're putting the necessary changes in place in our processes to make sure that you never have to experience similar circumstances in the future.
=Alain
0 comments:
Post a Comment