Author Topic: Server Restore  (Read 3284 times)

0 Members and 1 Guest are viewing this topic.

Offline Aaron Hulett

  • Administrator
  • Hero Member
  • *****
  • Posts: 1458
  • Schrödinger's cat walks into a bar... and doesn't.
    • View Profile
    • My Site
Server Restore
« on: November 09, 2010, 07:39:33 AM »
Opening this thread for discussion about the restore that took place just now that I posted about in the announcements section.  The main thing is posts and user registrations over the last 2 days were lost and you'll have to register again or post what's lost.  I apologize for the loss of the last two days - I know it impacts the site, especially the cleanup/help areas.

So what happened?  I'm not 100% sure.  The server started acting 'strange' and I started troubleshooting, and when I moved forward with a system restart, it never came back.  I opened a ticket to have the system reimaged and afterwards I restored a backup of the site.  The reimage means the box has the latest bits, and hopefully things will be stable going forward.

Let's hope this doesn't happen again (or if it does, not for a very, very long time).

//A

Offline roddy32

  • LzD Fallen Heroes
  • Hero Member
  • *****
  • Posts: 1075
    • View Profile
Re: Server Restore
« Reply #1 on: November 09, 2010, 10:26:56 AM »
Bummer Aaron, Hopefully it was just a glitch and I hope it doesn't happen again.
Microsoft MVP Consumer Security 2006 - 2012

Log'N'Rock Computer Security


Offline Temmu

  • The Assimilator
  • Hero Member
  • *****
  • Posts: 5404
    • View Profile
    • gooooooooogle
Re: Server Restore
« Reply #2 on: November 09, 2010, 05:17:39 PM »
huh.
not good.
glad it's back up, however.

Offline Aaron Hulett

  • Administrator
  • Hero Member
  • *****
  • Posts: 1458
  • Schrödinger's cat walks into a bar... and doesn't.
    • View Profile
    • My Site
Re: Server Restore
« Reply #3 on: November 10, 2010, 05:12:06 AM »
After a lot of digging and consulting, pretty much all signs point to the server entering an out-of-memory scenario, causing updates that were installing to fail, leading the system into a non-bootable state.  This continues to be apparent after the format/reimage as the system's available RAM has hovered roughly between 0 and 100 MBs.

I've ordered additional RAM for the server and while awaing its install I have disabled non-essential items, such as IRC.  Because the system was formatted, I'm not 100% sure this is what led to what happened (as the logs that would help, if any, were lost), but this looks to be promising.  I haven't heard when they'll install it yet but naturally there will be a period of downtime as the system is turned off and the RAM installed.

//A

Offline Aaron Hulett

  • Administrator
  • Hero Member
  • *****
  • Posts: 1458
  • Schrödinger's cat walks into a bar... and doesn't.
    • View Profile
    • My Site
Re: Server Restore
« Reply #4 on: November 10, 2010, 06:01:11 AM »
RAM is in. :)

Offline roddy32

  • LzD Fallen Heroes
  • Hero Member
  • *****
  • Posts: 1075
    • View Profile
Re: Server Restore
« Reply #5 on: November 10, 2010, 11:23:25 AM »
Let's hope for the best :)
Microsoft MVP Consumer Security 2006 - 2012

Log'N'Rock Computer Security