Michael D. Posted May 14, 2013 Report Share Posted May 14, 2013 Unfortunately the internet is a hostile place and there are a lot of individuals out there. There has been what is called a 'root escalation exploit' and in layman's terms it would permit any user to become an administrator and perform whatever commands they want [such as reading your data, wiping out the server, etc...]. This is an operating-system level exploit and not an issue we could have prevented ourselves and, as such, we have to rely upon our software vendors for patched versions of the operating systems. Due to the nature of this issue, as soon as a patch or updated operating system is available we will be rebooting all affected systems into the new kernel. While we don't anticipate the process to take more than 5 to 10 minutes per server, there is the possibility of unexpected issues such as a file system check on reboot that may take substantially longer. Currently the servers slated for reboots as soon as patches are available are as follows: Kobold, Jasmine, Icarus, Atlantis, and Boreas. When we are rebooting each server we will update this thread and if any issues arise, they will be posted here as well. We're also going to be sending out an email to all customers on the affected servers directing them to this thread as well. If you have any questions, by all means feel free to ask. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 14, 2013 Author Report Share Posted May 14, 2013 We are going to be rebooting the Atlantis VPS node right now to move into a newer kernel. The newer kernel is still vulnerable but there is a run-time patch we can apply to the new kernel to close the hole. We will keep this thread updated. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 14, 2013 Author Report Share Posted May 14, 2013 It never fails with OpenVZ -> There's an issue on boot that we're working on resolving. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 14, 2013 Author Report Share Posted May 14, 2013 I do not believe this issue with the Atlantis server is related to OpenVZ - we believe the server to be experiencing a hardware issue. We're working to resolve the issue as quickly as possible and will provide updates here as information becomes available. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Report Share Posted May 14, 2013 (edited) Our Jasmine server is being rebooted at this time. We expect it to be back online and responding normally within 5 to 10 minutes. Edited May 14, 2013 by Scott S Typos fixed Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Report Share Posted May 14, 2013 Our Jasmine server is being rebooted at this time. We expect it to be back online and responding normally within 5 to 10 minutes. The reboot of jasmine was delayed by approximately 10 minutes. Unfortunately it has not come back online as expected and we are currently investigating. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Report Share Posted May 14, 2013 We will now be posting updates specific to Jasmine and a new forum thread:http://forums.mddhosting.com/topic/871-jasmine-emergency-kernel-update-status/ Currently, we are still investigating the situation and have no new updates. Please post any questions regarding Jasmine in the new thread to avoid confusing this topic. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Report Share Posted May 14, 2013 We have started a new thread to track updates regarding Atlantis server. It can be found at:http://forums.mddhosting.com/topic/872-emergency-atlantis-kernel-update-status/ During the kernel update to prevent the zeroday exploit, we found a hardware issue that prevented us from bringing the server back online. We are working on the issue now. Please post any questions in the atlantis specific thread to prevent confusion in this forum thread. Quote Link to comment Share on other sites More sharing options...
Julia Posted May 14, 2013 Report Share Posted May 14, 2013 Any update on Kobold? Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Report Share Posted May 14, 2013 Any update on Kobold? Kobold (and icarus and boreas) are online and functioning normally. Updates have not been applied to them out of an abundance of caution due to problems with the patch on jasmine and atlantis. Quote Link to comment Share on other sites More sharing options...
Julia Posted May 14, 2013 Report Share Posted May 14, 2013 My site is showing as down on the following site http://www.downforeveryoneorjustme.com Should I submit a ticket? Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Report Share Posted May 14, 2013 My site is showing as down on the following site http://www.downforeveryoneorjustme.com Should I submit a ticket? If you are not located on atlantis or jasmine, and your site is currently verified as down, please submit a ticket. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Report Share Posted May 15, 2013 Jasmine is now online and responding normally. I'm also happy to say that it is running the newest kernel which is not vulnerable to this exploit. Atlantis suffered from a similar hardware failure and is still be repaired, but should be back online shortly. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Report Share Posted May 15, 2013 Atlantis is back online and individual VPSs should be online or nearly done booting at this time. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Author Report Share Posted May 15, 2013 We are about to reboot the Icarus server to apply this critical security patch. ETA 5~10 minutes. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Author Report Share Posted May 15, 2013 Icarus is now back on-line after 5 minutes of downtime. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Author Report Share Posted May 15, 2013 Boreas is being restarted. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Author Report Share Posted May 15, 2013 Boreas is back online. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Author Report Share Posted May 15, 2013 All servers but kobold have been secured. At least the last servers didn't face the problems of the first servers - they did exactly what they were supposed to which is restart in 5 minutes. We will be rebooting kobold as soon as our software vendor responsible for the software on kobold gets us a patched version. Quote Link to comment Share on other sites More sharing options...
kix766 Posted May 15, 2013 Report Share Posted May 15, 2013 will kobold be restarted 2day? Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Report Share Posted May 15, 2013 will kobold be restarted 2day? We are still waiting on a patched version from the software vendor. It will be installed as soon as it is available, at which time we will reboot the server. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 16, 2013 Author Report Share Posted May 16, 2013 It seems BetterLinux is not taking this security issue seriously. We're going to reboot the Kobold server now into a standard [secured] kernel due to the immense amount of risk all accounts on the Kobold server are at before this happens. This is the message I've sent to BetterLinux Management: If BetterLinux cannot apply a simple patch and provide an updated Kernel within 24 hours of a Zero-Day root escalation exploit - how can we possibly rely upon BetterLinux in a production environment? You're surely not going to compensate us if our server are compromised due to this exploit and data is destroyed.I realize BetterLinux has had a terrible time hitting deadlines, things that were 'coming next week' have taken 6+ months - but the patch is already out there - CentOS has already patched their kernels. OpenVZ has released a patched kernel, CloudLinux has released a patched kernel. KSplice has a patch for CentOS available.Where is the updated BetterLinux kernel?Don't tell me to change the "perf_event_paranoid" variable - this has already been worked around and won't keep the server secure. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 16, 2013 Author Report Share Posted May 16, 2013 We didn't anticipate the MySQL that comes with BetterLinux not being compatible with a standard kernel, so we may need to perform another reboot. We're working on bringing MySQL back online. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 16, 2013 Report Share Posted May 16, 2013 We didn't anticipate the MySQL that comes with BetterLinux not being compatible with a standard kernel, so we may need to perform another reboot. We're working on bringing MySQL back online. MySQL on kobold should be coming back online right now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 16, 2013 Author Report Share Posted May 16, 2013 We got this sorted out after also finding a small bug in the cPanel MySQL software manager. We're updating cPanel as to the steps we took so they can add some verbosity to their RPM management system. All services are back to normal. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.