Guild Launch News
Chicago Ventrilo Update
For most of our Ventrilo infrastructure in the United States, we use ColoCrossing. They have been a very good partner in helping to deliver the highest quality service to you. Under typical situations, the issue encountered yesterday would not have brought down access to Chicago as ColoCrossing has redundant links to two network providers. Unfortunately, this outage affected both of those network providers. Here is the official RFO (Reason For Outage) from ColoCrossing:
---------------------------------------------------------------------------------------
On Sunday afternoon ColoCrossing experienced a network event which was the result of carrier connectivity trouble. Our providers, Server Central and nLayer were the target of an advanced and carefully crafted Denial of Service attack. The malicious activity was focused on the core equipment of Server Central / nLayer and caused their routers to repeatedly drop IGP (internal) and BGP (external) routing protocols. Our carriers identified the attack and found that it exploited a loophole in the control-plane policer packet matching logic which normally protects against these types of attacks, allowing it to slip past these filters and cause the amount of impact that it did.
As an immediate solution our providers have applied additional packet filters at multiple network edges to block this type of traffic, and they are working with the equipment manufacturer to permanently close the loophole. At this time our carriers are confident that they have successfully blocked the attack and believe the network will be stable moving forward.
ColoCrossing regrets the inconvenience this has caused you and we are working expeditiously to ensure that future problems do not impact our customers. This was an all-hands-on-deck event, and our engineering team has been busy crafting a plan which will add a third carrier to our Chicago network. Once this third transit provider is added (which will be very soon) we will be able to effectively turn off our Server Central and nLayer connectivity should any future problems occur. It’s important to note that our existing network infrastructure was built to the highest standards and Server Central / nLayer have been an amazingly reliable partner for the last seven years. While this excellent track record does not excuse them for today’s problems it is certainly not cause to stop using their network. In the coming days you can look forward to being hosted on the most impressive network in Chicago, consisting of Server Central / nLayer and a third, still to be decided provider.
---------------------------------------------------------------------------------------
If anyone is still having issues accessing anything in Chicago, IL, please let us know ASAP and provide WinMTR data, along with your IP Address (the one your ISP has assigned you) so we can supply this information to our provider for them to troubleshoot further. You can download WinMTR from this URL: http://winmtr.sourceforge.net
-Stephen
---------------------------------------------------------------------------------------
On Sunday afternoon ColoCrossing experienced a network event which was the result of carrier connectivity trouble. Our providers, Server Central and nLayer were the target of an advanced and carefully crafted Denial of Service attack. The malicious activity was focused on the core equipment of Server Central / nLayer and caused their routers to repeatedly drop IGP (internal) and BGP (external) routing protocols. Our carriers identified the attack and found that it exploited a loophole in the control-plane policer packet matching logic which normally protects against these types of attacks, allowing it to slip past these filters and cause the amount of impact that it did.
As an immediate solution our providers have applied additional packet filters at multiple network edges to block this type of traffic, and they are working with the equipment manufacturer to permanently close the loophole. At this time our carriers are confident that they have successfully blocked the attack and believe the network will be stable moving forward.
ColoCrossing regrets the inconvenience this has caused you and we are working expeditiously to ensure that future problems do not impact our customers. This was an all-hands-on-deck event, and our engineering team has been busy crafting a plan which will add a third carrier to our Chicago network. Once this third transit provider is added (which will be very soon) we will be able to effectively turn off our Server Central and nLayer connectivity should any future problems occur. It’s important to note that our existing network infrastructure was built to the highest standards and Server Central / nLayer have been an amazingly reliable partner for the last seven years. While this excellent track record does not excuse them for today’s problems it is certainly not cause to stop using their network. In the coming days you can look forward to being hosted on the most impressive network in Chicago, consisting of Server Central / nLayer and a third, still to be decided provider.
---------------------------------------------------------------------------------------
If anyone is still having issues accessing anything in Chicago, IL, please let us know ASAP and provide WinMTR data, along with your IP Address (the one your ISP has assigned you) so we can supply this information to our provider for them to troubleshoot further. You can download WinMTR from this URL: http://winmtr.sourceforge.net
-Stephen