Michael D. Posted October 11, 2014 Report Share Posted October 11, 2014 Hello, We have to restart the Jasmine and Kobold servers for some important updates that we cannot apply without a reboot. Both servers have been online without fail for extended periods of time and although we are running KernelCare to update the kernel seamlessly it is necessary from time to time to still perform a reboot to actually move into a newer kernel. Jasmine has been online without reboot for 434 days and Kobold has been online for 172 days without a reboot. The reboot process for each should take no longer than 10 minutes. Quote Link to comment Share on other sites More sharing options...
Kevin Posted October 11, 2014 Report Share Posted October 11, 2014 When will you be rebooting? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 Here in a few minutes - we're actually working with cPanel regarding a stability issue we've been having on both servers and reboots are the next step in troubleshooting/resolving the issues. The server activity is extremely low at this point so the impact should be minimal. It only takes the server about 5 minutes to reboot and then 5 to 10 minutes for caches to fill up and things to get back up to speed/normal. I'll be keeping this thread up-to-date. Normally we would delay reboots but due to us working with our software vendor closely on this issue and it being a weekend we want to get this done as quickly as possible. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 Kobold is being rebooted now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 Kobold is coming back online now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 Jasmine is now being rebooted. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 Jasmine took longer to come down than expected - we'll be tweaking some settings regarding MySQL to help improve this in the future that should also have the side-effect of speeding up InnoDB access on Jasmine. Kobold is and has been back online and is still working on reflling it's caches to get back up to full speed. Jasmine is coming back up now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 The jasmine server had a kernel lock on boot - we're trying again - if it happens again we'll have to boot into an older version and get with CloudLinux support. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 Jasmine came up further this time but still inevitebly hung up. We're working on bring it back online. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 Jasmine is back online. We had to pull the power to the server physically after shutting it down and then plug it back in for it to boot. We've seen this problem before [i can't remember what causes it]. Thankfully having had this problem before we remembered this was a possible solution. We found out, in the process, that the CMOS battery in the Jasmine server is dead. We will be scheduling a window to replace the battery and we anticipate no issues with that maintenance. This will be at least 1 to 2 weeks out and we will email everybody letting you know when it will happen. It will take jasmine a few minutes to stabilize after this reboot. Let us know if you have any questions. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 I just checked Kobold and it's back to normal . Jasmine is still working on normalizing. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 Here are some graphs illustrating caches filling up and I/O stabilizing on the servers. Kobold is done but took longer than expected and Jasmine is still in the process of stabilizing fully. The below image is actually hosted on the Jasmine server . http://www.screen-shot.net/2014-10-11_13-55-36.png Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 Jasmine is continuing to stabilize. Things should be pretty responsive at this point but it's not quite back to normal just yet. http://www.screen-shot.net/2014-10-11_14-02-10.png Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 11, 2014 Author Report Share Posted October 11, 2014 Jasmine is now back to normal and actually appears to be performing a bit better than prior to the reboot. I'll close this thread out as resolved but if you have any questions about it you can post them here if there generic to the incident or open a ticket if they're specific to your account. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.