Author Topic: Today's website downtime  (Read 4659 times)

0 Members and 1 Guest are viewing this topic.

Offline Aaron Hulett

  • Administrator
  • Hero Member
  • *****
  • Posts: 1458
  • Schrödinger's cat walks into a bar... and doesn't.
    • View Profile
    • My Site
Today's website downtime
« on: September 12, 2006, 12:37:25 AM »
Sorry the site was unreachable earlier today.  Here's more information on why that happened.



At approx. 16:40hrs US CDT September 11th, we experienced an outage at Datacenter 1. This issue also affected inbound traffic directed to DC2 phase 2, via the DC1 backbones. This issue has been resolved, and connectivity is funtioning normally.

The source of this issue was a misconfiguration of the Fireslayer Dos Mitigation system. After reviewing logs we have discovered that an important safety measure was mistakenly removed from the system several days ago. Upon removal of fireslayer from a targeted IP this afternoon, a flawed firewall change was written to the edge routers. This change resulted in all inbound traffic at the DC1 edge to be blocked.

We aplogize to those customers who were affected by this issue. We will be reviewing this issue with the personnel involved. In additon we will be examining and implementing additonal safegaurds to the fireslayer system to prevent this type of issue from occuring in the future.

Brad Houlton
Sr. Network Engineer

Offline Corrine

  • The Mystical Rose
  • Administrator
  • Hero Member
  • *****
  • Posts: 20795
  • "Stronger than the past, united in our goal."
    • View Profile
    • Security Garden
Re: Today's website downtime
« Reply #1 on: September 12, 2006, 01:24:20 AM »
Thanks, Aaron.

That's nice communication from ev1 -- direct, honest, apologetic, and with a plan to prevent it happening in the future!

Take a walk through the "Security Garden" -- Where Everything is Coming up Roses!

Remember - A day without laughter is a day wasted.
May the wind sing to you and the sun rise in your heart.

Offline Temmu

  • The Assimilator
  • Hero Member
  • *****
  • Posts: 5404
    • View Profile
    • gooooooooogle
Re: Today's website downtime
« Reply #2 on: September 12, 2006, 11:00:34 AM »
huh.  thx for the update.

Offline GR@PH;<'S

  • Administrator
  • Hero Member
  • *****
  • Posts: 20125
    • View Profile
Re: Today's website downtime
« Reply #3 on: September 13, 2006, 01:39:30 PM »
Aaron Hulett - MSFT,

GR@PH;<'S   :Hammys pint:
press Enter then have a Brandy then if the problem is still there have another Brandy
Q: does it work
A: It does seem to for a few hours at least.