Scott Posted May 14, 2013 Report Share Posted May 14, 2013 (edited) During emergency maintenance to our jasmine server, we updated to a newer kernel that is not vulnerable to the current zero-day exploit. Unfortunately Jasmine did not reboot as expected and we are still investigating why. We will update this thread with information specific to Jasmine as we have new information for you. Update: 10:00pm - Marking this topic resolved. Edited May 15, 2013 by Scott S Resolved Quote Link to comment Share on other sites More sharing options...
jtslater Posted May 14, 2013 Report Share Posted May 14, 2013 Assuming you mean it did not reboot as expected? Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Author Report Share Posted May 14, 2013 Assuming you mean it did not reboot as expected? Yes, thank you. I've fixed the typo. I am using voice transcription to help with typing because my hand is currently in a brace for carpel tunnel. With the flood of tickets I did not double check the text as carefully as I should have. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Author Report Share Posted May 14, 2013 I don't have any new updates at this time, but we are still working on this and will update this thread as soon as we have more information for you. Quote Link to comment Share on other sites More sharing options...
Flix Posted May 14, 2013 Report Share Posted May 14, 2013 I wouldn't want to be Scott, Mike, or any other MDD admin right now. 1 Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Author Report Share Posted May 14, 2013 Jasmine is currently online, temporarily. We will be rebooting it again to reactivate CloudLinux. You may notice it rebooting several times as we continue to work on the issue and restore it to normal service. Quote Link to comment Share on other sites More sharing options...
samborac Posted May 14, 2013 Report Share Posted May 14, 2013 Wow. Hope this gets fixed ASAP, we have a back log of emails and are currently stuck at the office, because Time critical reports and exports need to be sent out before. Do have there is some means of compensation already APPROVED! Quote Link to comment Share on other sites More sharing options...
Scott Posted May 14, 2013 Author Report Share Posted May 14, 2013 Wow. Hope this gets fixed ASAPWe are working on it and want to see everything back to normal ASAP as well. we have a back log of emails and are currently stuck at the office, because Time critical reports and exports need to be sent out before. Do have there is some means of compensation already APPROVED! With the server temporarily online, you may be able to get these sent out right now. Hit your send/recieve button, attempt to resend the messages, or do whatever your email client requires when email is queued for sending. Quote Link to comment Share on other sites More sharing options...
Vman Posted May 15, 2013 Report Share Posted May 15, 2013 Is it correct to assume that the server is not back up yet? Our site is not loading and so I want to make sure it's not site specific. Thanks Quote Link to comment Share on other sites More sharing options...
ashkir Posted May 15, 2013 Report Share Posted May 15, 2013 Thanks for the updates. . I notified the non-profit offices I have on my hosting plan to not send sensitive documents right now and the fact that the mail server might be down. What will happen to emails being sent to something on my reseller? We're constantly sent mail with sensitive information. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Author Report Share Posted May 15, 2013 Is it correct to assume that the server is not back up yet? Our site is not loading and so I want to make sure it's not site specific. Thanks The site may be online sporadically while we continue to work on this. When service has been returned to normal (It hasn't yet), we will definiately make it clear that it is back online. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Author Report Share Posted May 15, 2013 Thanks for the updates. . I notified the non-profit offices I have on my hosting plan to not send sensitive documents right now and the fact that the mail server might be down. What will happen to emails being sent to something on my reseller? We're constantly sent mail with sensitive information. Someone on twitter beat you to this question: The messages should come through when everything is back online. https://twitter.com/MDDScott/status/334443298531975168 "Email servers will attempt to resend messages, typically for up to 72 hours, if they cannot connect to the receiving server." Quote Link to comment Share on other sites More sharing options...
Ryan Posted May 15, 2013 Report Share Posted May 15, 2013 Any update? The sites on this server still seem to be down... Quote Link to comment Share on other sites More sharing options...
FirestormIS Posted May 15, 2013 Report Share Posted May 15, 2013 Thanks for keeping us updated guys. Do you think we can expect it to be back up before midnight (CST)? Quote Link to comment Share on other sites More sharing options...
salty Posted May 15, 2013 Report Share Posted May 15, 2013 Is it correct to assume that the server is not back up yet? Our site is not loading and so I want to make sure it's not site specific. ThanksThis will show the server status:https://www.mddhosting.com/support/serverstatus.php Scott, I really do appreciate all the updates. While having a site go down sucks, it happens, but knowing exactly what's going on and why goes a long way to stem off any frustration. Great job. Quote Link to comment Share on other sites More sharing options...
Flix Posted May 15, 2013 Report Share Posted May 15, 2013 The bright side is that we now have a lot of new members on the forum! Viral marketing at its best. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Author Report Share Posted May 15, 2013 Any update? The sites on this server still seem to be down... I don't have any new information at this time. Thanks for keeping us updated guys. Do you think we can expect it to be back up before midnight (CST)? I hope so, but can't promise anything due to the nature of the issue. We could have this resolved in 10 minutes, or we could still be working in the morning. We are doing everything we can to get things back to normal ASAP. Bottom line: Still no ETA. (Sorry ) Quote Link to comment Share on other sites More sharing options...
FirestormIS Posted May 15, 2013 Report Share Posted May 15, 2013 I don't have any new information at this time. I hope so, but can't promise anything due to the nature of the issue. We could have this resolved in 10 minutes, or we could still be working in the morning. We are doing everything we can to get things back to normal ASAP. Bottom line: Still no ETA. (Sorry ) I figured that was probably the case. Good luck! Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Author Report Share Posted May 15, 2013 This will show the server status:https://www.mddhosting.com/support/serverstatus.phpGood tip Scott, I really do appreciate all the updates. While having a site go down sucks, it happens, but knowing exactly what's going on and why goes a long way to stem off any frustration. Great job.Thank you. The bright side is that we now have a lot of new members on the forum! Viral marketing at its best.This is definitely not my preference for marketing... but I suppose I would be remiss to not point out our twitter and facebook pages:twitter.com/mddhosting, twitter.com/mddscott, and facebook.com/mddhosting -- Of course, you will always get more information, faster, by using our forums. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Report Share Posted May 15, 2013 Hello everybody, I as well as two other administrators have been working to diagnose the issues with the Jasmine server and we've not been able to get the server to function properly. Everything is indicating hardware failure - it looks like processor failure. We're working as hard as we can to get service restored. Quote Link to comment Share on other sites More sharing options...
Jacqui Best Posted May 15, 2013 Report Share Posted May 15, 2013 1. This is unreal.. really a cloud server that can not be rolled back while you figure out what is wrong? Are you kidding me?2. It is not OK that you have no eta. I have client sites on here funneling thousands of dollars of leads to the email hosted on here. 3. I look unprofessional having recommended your server over oh lets say RACKSPACE who would never do a KERNEL update on a live server Quote Link to comment Share on other sites More sharing options...
Ryan Posted May 15, 2013 Report Share Posted May 15, 2013 Hello everybody, I as well as two other administrators have been working to diagnose the issues with the Jasmine server and we've not been able to get the server to function properly. Everything is indicating hardware failure - it looks like processor failure. We're working as hard as we can to get service restored.Better than hard drive I guess. Thanks for being on top of everything Mike. First time there has been an issue since I've been a customer for years and it seems like you guys are taking care of it as fast as you can. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Report Share Posted May 15, 2013 1. This is unreal.. really a cloud server that can not be rolled back while you figure out what is wrong? Are you kidding me?We run 'CloudLinux' which is a Linux Distribution, it is not a 'cloud server'. 2. It is not OK that you have no eta. I have client sites on here funneling thousands of dollars of leads to the email hosted on here.If we knew what the issue was we could provide an ETA and would have it resolved as quickly as possible. 3. I look unprofessional having recommended your server over oh lets say RACKSPACE who would never do a KERNEL update on a live serverKernel updates are standard fare, the issue is the new kernel didn't boot and upon rolling back to the old kernel it also would not boot. Data is intact and the array is working fine which indicates a hardware issue with either CPU or RAM most likely and processors are being swapped now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 15, 2013 Report Share Posted May 15, 2013 Processors have been swapped, booting it up now. Quote Link to comment Share on other sites More sharing options...
Scott Posted May 15, 2013 Author Report Share Posted May 15, 2013 Hi Jacqui, Thank you for your feedback. Please let me address each comment point by point: 1. This is unreal.. really a cloud server that can not be rolled back while you figure out what is wrong? Are you kidding me?When the kernel update failed, the first thing we did was attempt to roll back the kernel. Unfortunately, this also failed. As Michael explained in a post above yours, this points to a hardware (processor) failure and we are currently having the hardware in the server checked, and replaced if necessary. 2. It is not OK that you have no eta.Any ETA would be an absolute guess. We wouldn't feel comfortable advising you that the server will be online in any amount of time unless we were sure it would be back online by then. Under normal circumstances, and for most issues, we can give an ETA, but emergency maintenance is different. I have client sites on here funneling thousands of dollars of leads to the email hosted on here. 3. I look unprofessional having recommended your server over oh lets say RACKSPACE who would never do a KERNEL update on a live serverI'm sorry to hear about your companies image. As I am sure you can imagine, this doesn't look great for us either. The bottom line is that there was a critical zero day root escalation exploit being used in the wild, and it was imperative that we update our servers before any client site was compromised and used to take full access to the server. We would normally perform extensive testing prior to updating our servers kernel, but in this case we needed to act quickly before anything bad could happen. The kernel patches we applied came from a trusted source and we felt confident we could update safely. Unfortunately, we didn't anticipate hardware failure in one of our newer servers. You may wish to read more about the zero day vulnerability in our forum thread for that topic: http://forums.mddhosting.com/topic/870-multiple-servers-emergency-maintenance-reboots-to-close-operating-system-security-hole/ Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.