Today's website downtime

[Aaron Hulett]

Sorry the site was unreachable earlier today.  Here's more information on why that happened.

Aaron


From http://forums.ev1servers.net/showthread.php?t=63508

At approx. 16:40hrs US CDT September 11th, we experienced an outage at Datacenter 1. This issue also affected inbound traffic directed to DC2 phase 2, via the DC1 backbones. This issue has been resolved, and connectivity is funtioning normally.

The source of this issue was a misconfiguration of the Fireslayer Dos Mitigation system. After reviewing logs we have discovered that an important safety measure was mistakenly removed from the system several days ago. Upon removal of fireslayer from a targeted IP this afternoon, a flawed firewall change was written to the edge routers. This change resulted in all inbound traffic at the DC1 edge to be blocked.

We aplogize to those customers who were affected by this issue. We will be reviewing this issue with the personnel involved. In additon we will be examining and implementing additonal safegaurds to the fireslayer system to prevent this type of issue from occuring in the future.

Brad
__________________
Brad Houlton
Sr. Network Engineer
Ev1servers.net

[Corrine]

Thanks, Aaron.

That's nice communication from ev1 -- direct, honest, apologetic, and with a plan to prevent it happening in the future!

[Temmu]

huh.  thx for the update.

[GR@PH;<'S]

Aaron Hulett - MSFT,


GR@PH;<'S   :Hammys pint: