Jump to content
MDDHosting Forums

Multiple Servers - Emergency Maintenance Reboots to close Operating System Security Hole


Recommended Posts

Unfortunately the internet is a hostile place and there are a lot of individuals out there. There has been what is called a 'root escalation exploit' and in layman's terms it would permit any user to become an administrator and perform whatever commands they want [such as reading your data, wiping out the server, etc...]. This is an operating-system level exploit and not an issue we could have prevented ourselves and, as such, we have to rely upon our software vendors for patched versions of the operating systems.

 

Due to the nature of this issue, as soon as a patch or updated operating system is available we will be rebooting all affected systems into the new kernel. While we don't anticipate the process to take more than 5 to 10 minutes per server, there is the possibility of unexpected issues such as a file system check on reboot that may take substantially longer.

 

Currently the servers slated for reboots as soon as patches are available are as follows: Kobold, Jasmine, Icarus, Atlantis, and Boreas.

 

When we are rebooting each server we will update this thread and if any issues arise, they will be posted here as well.

 

We're also going to be sending out an email to all customers on the affected servers directing them to this thread as well.

 

If you have any questions, by all means feel free to ask.

Link to comment
Share on other sites

We are going to be rebooting the Atlantis VPS node right now to move into a newer kernel. The newer kernel is still vulnerable but there is a run-time patch we can apply to the new kernel to close the hole. We will keep this thread updated.

Link to comment
Share on other sites

I do not believe this issue with the Atlantis server is related to OpenVZ - we believe the server to be experiencing a hardware issue. We're working to resolve the issue as quickly as possible and will provide updates here as information becomes available.

Link to comment
Share on other sites

Our Jasmine server is being rebooted at this time. We expect it to be back online and responding normally within 5 to 10 minutes.

 

The reboot of jasmine was delayed by approximately 10 minutes. Unfortunately it has not come back online as expected and we are currently investigating.

Link to comment
Share on other sites

We have started a new thread to track updates regarding Atlantis server. It can be found at:

http://forums.mddhosting.com/topic/872-emergency-atlantis-kernel-update-status/

 

During the kernel update to prevent the zeroday exploit, we found a hardware issue that prevented us from bringing the server back online. We are working on the issue now. Please post any questions in the atlantis specific thread to prevent confusion in this forum thread.

Link to comment
Share on other sites

Jasmine is now online and responding normally. I'm also happy to say that it is running the newest kernel which is not vulnerable to this exploit.

 

Atlantis suffered from a similar hardware failure and is still be repaired, but should be back online shortly.

Link to comment
Share on other sites

All servers but kobold have been secured. At least the last servers didn't face the problems of the first servers - they did exactly what they were supposed to which is restart in 5 minutes.

 

We will be rebooting kobold as soon as our software vendor responsible for the software on kobold gets us a patched version.

Link to comment
Share on other sites

It seems BetterLinux is not taking this security issue seriously. We're going to reboot the Kobold server now into a standard [secured] kernel due to the immense amount of risk all accounts on the Kobold server are at before this happens.

 

This is the message I've sent to BetterLinux Management:

 

 

If BetterLinux cannot apply a simple patch and provide an updated Kernel within 24 hours of a Zero-Day root escalation exploit - how can we possibly rely upon BetterLinux in a production environment? You're surely not going to compensate us if our server are compromised due to this exploit and data is destroyed.

I realize BetterLinux has had a terrible time hitting deadlines, things that were 'coming next week' have taken 6+ months - but the patch is already out there - CentOS has already patched their kernels. OpenVZ has released a patched kernel, CloudLinux has released a patched kernel. KSplice has a patch for CentOS available.

Where is the updated BetterLinux kernel?

Don't tell me to change the "perf_event_paranoid" variable - this has already been worked around and won't keep the server secure.
Link to comment
Share on other sites

We didn't anticipate the MySQL that comes with BetterLinux not being compatible with a standard kernel, so we may need to perform another reboot. We're working on bringing MySQL back online.

 

MySQL on kobold should be coming back online right now.

Link to comment
Share on other sites

We got this sorted out after also finding a small bug in the cPanel MySQL software manager. We're updating cPanel as to the steps we took so they can add some verbosity to their RPM management system.

 

All services are back to normal.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...