Jump to content
MDDHosting Forums

Jasmine Server - Kernel Panic resulting in Corrupted Data


Recommended Posts

At this point I'm running a file system check, it appears that this particular kernel panic may have resulted in a lot of garbage data/file system corruption. If the file system check doesn't resolve this, and a couple of other things that we have that we're going to try as well - we may need to restore this server from backups. We're doing everything we can to avoid that, and will update you as there is information to provide.
Link to comment
Share on other sites

Still in progress but the longer it runs, the less confident I am that it will fix the issue. As soon as it's completed, you will know here. I assure you I want it done as quickly as possible, and would make it go faster if I could.
Link to comment
Share on other sites

The server booted up, but there are a large number of system files damaged and, possibly, user files damaged. We are still working on fully resolving this, and will update you when we're done. In the meantime - your sites should be online. If they are not online, do open a ticket and/or update your ticket (if you already opened one) so we can address issues on a one-to-one basis.
Link to comment
Share on other sites

I've put up a temporary copy of the MySQL data in another location so at least most sites will be working while I restore MySQL data from our latest backup. Once this is done, MySQL and InnoDB should start up with yesterday's data. At that point, we can selectively move databases over from the current data set (i.e. data from just before the server crash).

 

This is all assuming we can get all of the file system issues worked out such as damaged system files, dependencies, etc. We may end up having to do a bare metal restoration, but we're doing everything we can to avoid it.

Link to comment
Share on other sites

InnoDB is online, however, most InnoDB tables are damaged beyond repair and, as such, will need restored from our backups. We're doing our best to pro-actively do this. The server will be rebooted at some point in the next few hours to move to a newer/stable kernel. There is the small possibility that on this reboot, the system requires a file system check or that the system would refuse to boot again, resulting in more downtime than expected for a normal reboot.

 

We'll be keeping this thread up-to-date.

Link to comment
Share on other sites

I believe everything, at this point, should be back to normal. We did end up having to resort to restoring databases from our latest restoration point (12/02/2013 at 12:30 AM). If you experience any issues at all, do please open a support ticket so we can address them.
Link to comment
Share on other sites

Thankfully you guys have been informing us of the situation each step of the way. I was able to backup our database an hour before you had to restore them from yesterday's backups. Just restored from my backup a minute ago and only an hour's worth of activity may have been lost (if there was any this time of night).

 

I don't know of any other host that would keep us informed like that. Nor one that would actually have a backup that was only a day old to restore from.

 

Thank you!

Link to comment
Share on other sites

Some were lucky, and were able to retain their old data but the majority were not so lucky. We still have the corrupted data and are are working to restore individual databases as we are able, but there's admittedly only a select few that we've been able to restore fully thus far.
Link to comment
Share on other sites

 Share

×
×
  • Create New...