Guild Launch News
DOWNTIME: Harddrive Failure on Session Database
At around 6:30 PM we had 2 harddrive failures on two different machines. We are looking into how that happened, but harddrives have been known to come in bad batches and these servers were ordered around the same time as one another.
We have recovered nearly full functionality on the primary Guild Launch site since the database server that the drives failed on only houses session data and full text search for those services. We've moved the session data to the primary Guild Launch DB server at this time.
Forum searching will be offline until the secondary DB is restored and we can reload the full text indexes.
At this time Advertising and Sigs are still down until we restore those databases.
Two hard drive failures is truly a lightning strike. At this time while the downtime is very unfortunate we have enough redundancy, load balanced servers, etc. that we are able to move services around quickly and restore service with minimal downtime. We will take the lessons learned from this and continue to improve our processes.
We appreciate your patience while we get everything up to 100% and we will keep you all updated on the progress.
-Stephen
UPDATE 10:30 PM 11/05/2008
Sigs, Thumper and the Advertising Server are up and running. Primary GL functionality is fully functional except for forum search. Forum search will be restored in the morning when there is less traffic due to the impact of index rebuilding.
UPDATE 8:15 AM 11/06/2008
We are beginning the process of restoring text searches. The indexes for all site should be rebuilt around lunch time EST. Individual site searches will come online as their individual indexes get rebuilt throughout the morning.
UPDATE 10:00 PM 11/06/2008
All search indexes have been rebuilt at this time. However, some indexes are a bit out of date. We will be running a catchup process next Wednesday to get them 100% in sync.
We have recovered nearly full functionality on the primary Guild Launch site since the database server that the drives failed on only houses session data and full text search for those services. We've moved the session data to the primary Guild Launch DB server at this time.
Forum searching will be offline until the secondary DB is restored and we can reload the full text indexes.
At this time Advertising and Sigs are still down until we restore those databases.
Two hard drive failures is truly a lightning strike. At this time while the downtime is very unfortunate we have enough redundancy, load balanced servers, etc. that we are able to move services around quickly and restore service with minimal downtime. We will take the lessons learned from this and continue to improve our processes.
We appreciate your patience while we get everything up to 100% and we will keep you all updated on the progress.
-Stephen
UPDATE 10:30 PM 11/05/2008
Sigs, Thumper and the Advertising Server are up and running. Primary GL functionality is fully functional except for forum search. Forum search will be restored in the morning when there is less traffic due to the impact of index rebuilding.
UPDATE 8:15 AM 11/06/2008
We are beginning the process of restoring text searches. The indexes for all site should be rebuilt around lunch time EST. Individual site searches will come online as their individual indexes get rebuilt throughout the morning.
UPDATE 10:00 PM 11/06/2008
All search indexes have been rebuilt at this time. However, some indexes are a bit out of date. We will be running a catchup process next Wednesday to get them 100% in sync.
Last edited by GL_Support on Sat Nov 08, 2008 3:11 pm; edited 3 times in total
Can you confirm that this is affecting the Ventrilo servers?
Vent Server is down and Status shows as Inactive on Guild Launch Subscriptions but shows as Active on PayPal under subscriptions.
Vent Server is down and Status shows as Inactive on Guild Launch Subscriptions but shows as Active on PayPal under subscriptions.
ahuatamahal wrote:
This isn't effecting Ventrilo servers at all. What is your Ventrilo HOST and PORT?
-Stephen
Can you confirm that this is affecting the Ventrilo servers?
Vent Server is down and Status shows as Inactive on Guild Launch Subscriptions but shows as Active on PayPal under subscriptions.
Vent Server is down and Status shows as Inactive on Guild Launch Subscriptions but shows as Active on PayPal under subscriptions.
This isn't effecting Ventrilo servers at all. What is your Ventrilo HOST and PORT?
-Stephen
GL_Support wrote:
Meh, it's up now...but my subscriptions page still says inactive. I also couldn't log into the control panel earlier. Was down for about 30 minutes that I was aware of.
Very Weird - on 10/23 I upgraded from 15 slot to 30 slot.
Ventrilo Subscriptions
PayPal Subscription ID: S-64J423732B743341K
Signup Date: 2008/11/05
Status: INACTIVE
...truncated...
Payment
Guild Launch Transaction ID: 25331
Trans Date: 2008/09/03
Amount: 5.75 USD
Payment
Guild Launch Transaction ID: 28401
Trans Date: 2008/10/03
Amount: 5.75 USD
Subscription Cancellation
Guild Launch Transaction ID: 30389
Trans Date: 2008/10/23
Subscription End of Term
Guild Launch Transaction ID: 31369
Trans Date: 2008/11/03
ahuatamahal wrote:
This isn't effecting Ventrilo servers at all. What is your Ventrilo HOST and PORT?
-Stephen
Can you confirm that this is affecting the Ventrilo servers?
Vent Server is down and Status shows as Inactive on Guild Launch Subscriptions but shows as Active on PayPal under subscriptions.
Vent Server is down and Status shows as Inactive on Guild Launch Subscriptions but shows as Active on PayPal under subscriptions.
This isn't effecting Ventrilo servers at all. What is your Ventrilo HOST and PORT?
-Stephen
Meh, it's up now...but my subscriptions page still says inactive. I also couldn't log into the control panel earlier. Was down for about 30 minutes that I was aware of.
Very Weird - on 10/23 I upgraded from 15 slot to 30 slot.
Ventrilo Subscriptions
PayPal Subscription ID: S-64J423732B743341K
Signup Date: 2008/11/05
Status: INACTIVE
...truncated...
Payment
Guild Launch Transaction ID: 25331
Trans Date: 2008/09/03
Amount: 5.75 USD
Payment
Guild Launch Transaction ID: 28401
Trans Date: 2008/10/03
Amount: 5.75 USD
Subscription Cancellation
Guild Launch Transaction ID: 30389
Trans Date: 2008/10/23
Subscription End of Term
Guild Launch Transaction ID: 31369
Trans Date: 2008/11/03
same thing happened to mine (ventrilo account) although not from upgrading, steve knows how to fix it .....
what luck that they both died at once :S even being from the same batch...
make sure you check all of them and blast the supplier and return ALL for replacement.
dont want the rest following suit in a month.
what luck that they both died at once :S even being from the same batch...
make sure you check all of them and blast the supplier and return ALL for replacement.
dont want the rest following suit in a month.