Server Restore

[Aaron Hulett]

Opening this thread for discussion about the restore that took place just now that I posted about in the announcements section.  The main thing is posts and user registrations over the last 2 days were lost and you'll have to register again or post what's lost.  I apologize for the loss of the last two days - I know it impacts the site, especially the cleanup/help areas.

So what happened?  I'm not 100% sure.  The server started acting 'strange' and I started troubleshooting, and when I moved forward with a system restart, it never came back.  I opened a ticket to have the system reimaged and afterwards I restored a backup of the site.  The reimage means the box has the latest bits, and hopefully things will be stable going forward.

Let's hope this doesn't happen again (or if it does, not for a very, very long time).

//A

[roddy32]

Bummer Aaron, Hopefully it was just a glitch and I hope it doesn't happen again.

[Temmu]

huh.
not good.
glad it's back up, however.

[Aaron Hulett]

After a lot of digging and consulting, pretty much all signs point to the server entering an out-of-memory scenario, causing updates that were installing to fail, leading the system into a non-bootable state.  This continues to be apparent after the format/reimage as the system's available RAM has hovered roughly between 0 and 100 MBs.

I've ordered additional RAM for the server and while awaing its install I have disabled non-essential items, such as IRC.  Because the system was formatted, I'm not 100% sure this is what led to what happened (as the logs that would help, if any, were lost), but this looks to be promising.  I haven't heard when they'll install it yet but naturally there will be a period of downtime as the system is turned off and the RAM installed.

//A

[Aaron Hulett]

RAM is in. :)

[roddy32]

Let's hope for the best :)