Guild Launch News
Outage - 6/30/2007 - 4:30am
What Happened?
As many of you know we recently upgraded our server infrastructure to improve performance and stability. These changes to the infrastructure over the last week have required some settings which we had previously stabilized to be incorrect for our current load and for some of the software we have running such as the Signature Generator. This morning we had unexpected outage around 4:30am Eastern. We have been adjusting these settings to stabilize the servers.
These issues *will* be resolved shortly, we are very close to the final settings and we are monitoring things closely.
What are you doing in the meantime?
We have added more checks with our hosting facility so that their internal monitors will track this issue and respond more quickly and we have again adjusted settings along our planned path to respond to this issue.
Didn't you test this stuff?
We ran load and configuration testing internally in-house for close to a week. Which is why we know the new software performs so much better than the old one. Sometimes real world load can change things subtly.
What are the long term repercussions?
In the next few days we will have the settings for the new hardware finalized and we will be fully on both a faster and more stable system.
Can we expect more downtime?
We believe that we have all scenarios covered with monitors now and no downtime related to this issue should last more than a few minutes and there should be no more downtime related to this after the next few days.
We apologize for the downtime this morning, and your patience and understanding is appreciated. Michael, Monica, Mike and I definitely want this to be resolved as much as you so we can get back to developing our awesome features full time.
-Stephen
As many of you know we recently upgraded our server infrastructure to improve performance and stability. These changes to the infrastructure over the last week have required some settings which we had previously stabilized to be incorrect for our current load and for some of the software we have running such as the Signature Generator. This morning we had unexpected outage around 4:30am Eastern. We have been adjusting these settings to stabilize the servers.
These issues *will* be resolved shortly, we are very close to the final settings and we are monitoring things closely.
What are you doing in the meantime?
We have added more checks with our hosting facility so that their internal monitors will track this issue and respond more quickly and we have again adjusted settings along our planned path to respond to this issue.
Didn't you test this stuff?
We ran load and configuration testing internally in-house for close to a week. Which is why we know the new software performs so much better than the old one. Sometimes real world load can change things subtly.
What are the long term repercussions?
In the next few days we will have the settings for the new hardware finalized and we will be fully on both a faster and more stable system.
Can we expect more downtime?
We believe that we have all scenarios covered with monitors now and no downtime related to this issue should last more than a few minutes and there should be no more downtime related to this after the next few days.
We apologize for the downtime this morning, and your patience and understanding is appreciated. Michael, Monica, Mike and I definitely want this to be resolved as much as you so we can get back to developing our awesome features full time.
-Stephen