Author Topic: Server Restore  (Read 651 times)

0 Members and 1 Guest are viewing this topic.

Offline Aaron Hulett [MSFT]

  • Web Server Manager
  • Administrator
  • Hero Member
  • *****
  • Posts: 1098
  • I take the bus!
    • Microsoft Corporation
Server Restore
« on: November 09, 2010, 08:39:33 AM »
Opening this thread for discussion about the restore that took place just now that I posted about in the announcements section.  The main thing is posts and user registrations over the last 2 days were lost and you'll have to register again or post what's lost.  I apologize for the loss of the last two days - I know it impacts the site, especially the cleanup/help areas.

So what happened?  I'm not 100% sure.  The server started acting 'strange' and I started troubleshooting, and when I moved forward with a system restart, it never came back.  I opened a ticket to have the system reimaged and afterwards I restored a backup of the site.  The reimage means the box has the latest bits, and hopefully things will be stable going forward.

Let's hope this doesn't happen again (or if it does, not for a very, very long time).

//A
Aaron Hulett | Malware Protection Center | Microsoft Corporation
This post is provided "AS IS" without warranty, and confers no rights.

Offline roddy32

  • Update Moderator
  • Hero Member
  • *****
  • Posts: 1165
Re: Server Restore
« Reply #1 on: November 09, 2010, 11:26:56 AM »
Bummer Aaron, Hopefully it was just a glitch and I hope it doesn't happen again.
Microsoft MVP Consumer Security 2006 - 2012

Log'N'Rock Computer Security


Offline Temmu

  • The Assimilator
  • Hero Member
  • *****
  • Posts: 4999
    • gooooooooogle
Re: Server Restore
« Reply #2 on: November 09, 2010, 06:17:39 PM »
huh.
not good.
glad it's back up, however.
Linux Registered User #433481

[ t h i s . s p a c e . f o r . r e n t ] ~ debian ~ ubuntu ~ pclinuxos ~ xp ~ opera ~ firefox ~ keyboard ~ mouse ~ color monitor ~

click here

Offline Aaron Hulett [MSFT]

  • Web Server Manager
  • Administrator
  • Hero Member
  • *****
  • Posts: 1098
  • I take the bus!
    • Microsoft Corporation
Re: Server Restore
« Reply #3 on: November 10, 2010, 06:12:06 AM »
After a lot of digging and consulting, pretty much all signs point to the server entering an out-of-memory scenario, causing updates that were installing to fail, leading the system into a non-bootable state.  This continues to be apparent after the format/reimage as the system's available RAM has hovered roughly between 0 and 100 MBs.

I've ordered additional RAM for the server and while awaing its install I have disabled non-essential items, such as IRC.  Because the system was formatted, I'm not 100% sure this is what led to what happened (as the logs that would help, if any, were lost), but this looks to be promising.  I haven't heard when they'll install it yet but naturally there will be a period of downtime as the system is turned off and the RAM installed.

//A
Aaron Hulett | Malware Protection Center | Microsoft Corporation
This post is provided "AS IS" without warranty, and confers no rights.

Offline Aaron Hulett [MSFT]

  • Web Server Manager
  • Administrator
  • Hero Member
  • *****
  • Posts: 1098
  • I take the bus!
    • Microsoft Corporation
Re: Server Restore
« Reply #4 on: November 10, 2010, 07:01:11 AM »
RAM is in. :)
Aaron Hulett | Malware Protection Center | Microsoft Corporation
This post is provided "AS IS" without warranty, and confers no rights.

Offline roddy32

  • Update Moderator
  • Hero Member
  • *****
  • Posts: 1165
Re: Server Restore
« Reply #5 on: November 10, 2010, 12:23:25 PM »
Let's hope for the best :)
Microsoft MVP Consumer Security 2006 - 2012

Log'N'Rock Computer Security