LandzDown Forum

Miscellaneous => LandzDown Lounge => Topic started by: Aaron Hulett on September 12, 2006, 01:37:25 AM

Title: Today's website downtime
Post by: Aaron Hulett on September 12, 2006, 01:37:25 AM
Sorry the site was unreachable earlier today.  Here's more information on why that happened.

Aaron


From http://forums.ev1servers.net/showthread.php?t=63508

QuoteAt approx. 16:40hrs US CDT September 11th, we experienced an outage at Datacenter 1. This issue also affected inbound traffic directed to DC2 phase 2, via the DC1 backbones. This issue has been resolved, and connectivity is funtioning normally.

The source of this issue was a misconfiguration of the Fireslayer Dos Mitigation system. After reviewing logs we have discovered that an important safety measure was mistakenly removed from the system several days ago. Upon removal of fireslayer from a targeted IP this afternoon, a flawed firewall change was written to the edge routers. This change resulted in all inbound traffic at the DC1 edge to be blocked.

We aplogize to those customers who were affected by this issue. We will be reviewing this issue with the personnel involved. In additon we will be examining and implementing additonal safegaurds to the fireslayer system to prevent this type of issue from occuring in the future.

Brad
__________________
Brad Houlton
Sr. Network Engineer
Ev1servers.net

Title: Re: Today's website downtime
Post by: Corrine on September 12, 2006, 02:24:20 AM
Thanks, Aaron.

That's nice communication from ev1 -- direct, honest, apologetic, and with a plan to prevent it happening in the future!
Title: Re: Today's website downtime
Post by: Temmu on September 12, 2006, 12:00:34 PM
huh.  thx for the update.
Title: Re: Today's website downtime
Post by: GR@PH;<'S on September 13, 2006, 02:39:30 PM
Aaron Hulett - MSFT,
(https://www.landzdown.com/proxy.php?request=http%3A%2F%2Fsmilies.sofrayt.com%2Ffsc%2Fgentleman.gif&hash=6c1090d9e743a45850d1ab96b5ae1394649c9d02)

GR@PH;<'S   :Hammys pint: