Michael D. Posted January 3, 2013 Report Share Posted January 3, 2013 The Jasmine server has experienced a Kernel panic, forcing a reset. We'll update this thread with more details when we have them. If you wish to see the panic: http://www.screen-sh...02_22-46-32.png Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 Upon reboot, the server is panicing on kernel load as well, this may take longer than a few minutes to resolve, but we're working on it. Quote Link to comment Share on other sites More sharing options...
Scott Posted January 3, 2013 Report Share Posted January 3, 2013 We are still working on this issue, but do not have an ETA at this time. We will post more details as soon as possible, but our current focus is on getting the server back online. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 At this point I'm running a file system check, it appears that this particular kernel panic may have resulted in a lot of garbage data/file system corruption. If the file system check doesn't resolve this, and a couple of other things that we have that we're going to try as well - we may need to restore this server from backups. We're doing everything we can to avoid that, and will update you as there is information to provide. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 File system check still in progress, nothing new to report at this time. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 File system check is still in progress, unfortunately it does not provide an ETA but generally it's between 30 and 120 minutes from start to finish. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 Still in progress but the longer it runs, the less confident I am that it will fix the issue. As soon as it's completed, you will know here. I assure you I want it done as quickly as possible, and would make it go faster if I could. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 The server is rebooting now, the FSCK just finished. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 The server appears to be booting up properly. We do have backups if any customer data has been damaged. There will be another reboot to move to a known-stable kernel shortly. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 The server booted up, but there are a large number of system files damaged and, possibly, user files damaged. We are still working on fully resolving this, and will update you when we're done. In the meantime - your sites should be online. If they are not online, do open a ticket and/or update your ticket (if you already opened one) so we can address issues on a one-to-one basis. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 InnoDB has been disabled to bring MySQL up, if your site uses InnoDB you will get database errors - we are still working to restore the server 100% as there was a lot of file system corruption. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 MySQL is back offline at this time as we work to restore full functionality. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 I've put up a temporary copy of the MySQL data in another location so at least most sites will be working while I restore MySQL data from our latest backup. Once this is done, MySQL and InnoDB should start up with yesterday's data. At that point, we can selectively move databases over from the current data set (i.e. data from just before the server crash). This is all assuming we can get all of the file system issues worked out such as damaged system files, dependencies, etc. We may end up having to do a bare metal restoration, but we're doing everything we can to avoid it. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 Note: InnoDB is still unavailable, we'll update you once it's back. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 InnoDB is online, however, most InnoDB tables are damaged beyond repair and, as such, will need restored from our backups. We're doing our best to pro-actively do this. The server will be rebooted at some point in the next few hours to move to a newer/stable kernel. There is the small possibility that on this reboot, the system requires a file system check or that the system would refuse to boot again, resulting in more downtime than expected for a normal reboot. We'll be keeping this thread up-to-date. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 We are still working on this, there is a lot of data to repair so it's taking time. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 The server is now being rebooted. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 I believe everything, at this point, should be back to normal. We did end up having to resort to restoring databases from our latest restoration point (12/02/2013 at 12:30 AM). If you experience any issues at all, do please open a support ticket so we can address them. Quote Link to comment Share on other sites More sharing options...
drathe Posted January 3, 2013 Report Share Posted January 3, 2013 Thankfully you guys have been informing us of the situation each step of the way. I was able to backup our database an hour before you had to restore them from yesterday's backups. Just restored from my backup a minute ago and only an hour's worth of activity may have been lost (if there was any this time of night). I don't know of any other host that would keep us informed like that. Nor one that would actually have a backup that was only a day old to restore from. Thank you! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 3, 2013 Author Report Share Posted January 3, 2013 Some were lucky, and were able to retain their old data but the majority were not so lucky. We still have the corrupted data and are are working to restore individual databases as we are able, but there's admittedly only a select few that we've been able to restore fully thus far. Quote Link to comment Share on other sites More sharing options...
joshualoy Posted January 4, 2013 Report Share Posted January 4, 2013 Is this related to bugs in R1soft or Cloudlinux at all? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 5, 2013 Author Report Share Posted January 5, 2013 Indeed, an issue with the CloudLinux kernel. It had been stable for us for an extended period of time and then decided to randomly panic. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.