Scott Posted May 14, 2013 Report Share Posted May 14, 2013 (edited) During emergency maintenance to our atlantis server, we updated to a newer kernel that is not vulnerable to the current zero-day exploit. Unfortunately we found a seperate hardware issue affecting atlantis that prevented us from bringing it back online. We are currently testing and replacing affected hardware in this server. We will post any updates regarding this hardware issue with atlantis in this forum thread. Update: 9:56pm - This issue is now resolved. Edited May 15, 2013 by Scott S Resolved Quote Link to comment Share on other sites More sharing options...
scottcocoabeach Posted May 14, 2013 Report Share Posted May 14, 2013 Is data expected to be affected? Can you provide a estimated time frame to be up and running if you know one. Thanks. Quote Link to comment Share on other sites More sharing options...
zeratul Posted May 14, 2013 Report Share Posted May 14, 2013 Same here. I need an ETA please, my clients are calling... Thanks. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Author Report Share Posted May 14, 2013 Is data expected to be affected? Can you provide a estimated time frame to be up and running if you know one. Thanks. We have no reason to expect there will be any data loss. Additionally all VPS servers are back up nightly in our R1Soft backup system, so you are protected either way (but there is no reason to think that we will need to restore backups at this time). Same here. I need an ETA please, my clients are calling... Thanks. Are technicians and administrators are still working at this time, but unfortunately we do not have an ETA. Quote Link to comment Share on other sites More sharing options...
le.gentleman Posted May 14, 2013 Report Share Posted May 14, 2013 What exactly is the hardware issue? Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Author Report Share Posted May 14, 2013 What exactly is the hardware issue? I don't currently have any details on the specific hardware issue to provide. We are still conducting tests on the server and its hardware. Quote Link to comment Share on other sites More sharing options...
Flix Posted May 14, 2013 Report Share Posted May 14, 2013 What exactly is the hardware issue?Hard to tell when you're 900 miles from the actual hardware... Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Author Report Share Posted May 14, 2013 Hard to tell when you're 900 miles from the actual hardware... We have people on site and working with our hardware as we speak. Quote Link to comment Share on other sites More sharing options...
cvos Posted May 15, 2013 Report Share Posted May 15, 2013 Is there an eta on Atlantis. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Author Report Share Posted May 15, 2013 Is there an eta on Atlantis. Not at this time. As soon as we have it, you you will have it (posted here). Quote Link to comment Share on other sites More sharing options...
frankacter Posted May 15, 2013 Report Share Posted May 15, 2013 Are there any contingencies being considered and a timeline for when that would be considered (ie. restoring to different hardware, etc) Quote Link to comment Share on other sites More sharing options...
Julia Posted May 15, 2013 Report Share Posted May 15, 2013 Any further updates on this? Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Author Report Share Posted May 15, 2013 Are there any contingencies being considered and a timeline for when that would be considered (ie. restoring to different hardware, etc) Due to the amount of data, it would take a long while to fully restore the VPS server to new hardware. We may begin discussing it soon, but we are still trying to isolate the hardware failure in the server and swap out the faulty hardware instead. Quote Link to comment Share on other sites More sharing options...
frankacter Posted May 15, 2013 Report Share Posted May 15, 2013 Thanks Scott. Let me know when I need to send pizza and caffeine care packages Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Report Share Posted May 15, 2013 Data is intact, we're not restoring data to new hardware. The processors in the server have failed and we're swapping them both now as I type this. The server should be back online in 10 to 15 minutes [rough ETA]. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Report Share Posted May 15, 2013 Atlantis is booting up with new processors perfectly. VPS are starting up one at a time [that's how OpenVZ does it to keep from overloading the node and crashing it out on boot]. As soon as all are started up I'll update this thread. Bear in mind after your VPS is online it may take 5~15 minutes for everything to 'level out' and get back to normal. Quote Link to comment Share on other sites More sharing options...
le.gentleman Posted May 15, 2013 Report Share Posted May 15, 2013 Excellent, let us know once it is actually up and running, thanks. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Author Report Share Posted May 15, 2013 Excellent, let us know once it is actually up and running, thanks. Everything should be booting up now, and likely finished within a minute or two. After that, the server may need 5-10 minutes to catch up from the mountain of requests that hit it while booting up. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Author Report Share Posted May 15, 2013 I spoke a bit too soon. Some individual VPSs are still booting up. About 80% are finished at this time and the rest should be up and running shortly. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Author Report Share Posted May 15, 2013 Excellent, let us know once it is actually up and running, thanks.Everything is up and running. (Or should be... Open a support ticket if you're having issues) Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Report Share Posted May 15, 2013 Additional details on this issue, how it was handled, and why can be found in this post: http://forums.mddhosting.com/topic/871-jasmine-emergency-kernel-update-status/?p=4429 I'm linking to it rather than re-posting it just to be concise and keep the discussion in one place. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Report Share Posted May 15, 2013 In my efforts to get the server back online as quickly as possible and likely due to exhaustion/stress I selected the incorrect kernel on the last boot. I'm going to issue a reboot right now so your data is secure. Everybody will be back online as soon as possible [i estimate 5~15 minutes]. I am so sorry to have to do this but the potential consequences of not doing it [such as your VPS being wiped/deleted by a hacker or your data stolen] are too grave to put this off. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Report Share Posted May 15, 2013 The node is back online, and VPS are starting back up - again, staggered so that the server doesn't get entirely overloaded. ETA 5~10 min for all VPS to be up if I'm estimating correctly and the server is 100% Secure from the new exploit. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.