Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Unfortunately the news is not good. We're going to have to conduct another restoration to one day back into the past. The restoration time is likely to take about the same amount of time (approximately 7 hours). I'll keep this thread updated. Quote Link to comment Share on other sites More sharing options...
jack Posted June 21, 2011 Report Share Posted June 21, 2011 Unfortunately the news is not good. We're going to have to conduct another restoration to one day back into the past. The restoration time is likely to take about the same amount of time (approximately 7 hours). I'll keep this thread updated. What was the problem with the first restoration? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 What was the problem with the first restoration?The file system is reporting issues even after a restoration, we're going to go one more day back and do another restoration. If the file system comes back corrupted on a day further back that means that we're going to need to go about doing the restorations manually which is an extremely labor intensive and time intensive process. I'll keep the thread updated. Quote Link to comment Share on other sites More sharing options...
jian Posted June 21, 2011 Report Share Posted June 21, 2011 Mike first of all thanks for the updates. You mentioned that you have to do another restore, can you shed some light as to why? Has the restore on the identical new hardware also failed as well for this restore? What happens if this restore fails again? Thanks, keep up the fight! Quote Link to comment Share on other sites More sharing options...
Themeltingclock Posted June 21, 2011 Report Share Posted June 21, 2011 Unfortunately the news is not good. We're going to have to conduct another restoration to one day back into the past. The restoration time is likely to take about the same amount of time (approximately 7 hours). I'll keep this thread updated.:/ Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 The file system is still reporting errors, we're going to try going back one additional day and cross our fingers that the file system is clean. If not, we'll simply need to run a file system check again after the restoration and get to work on fixing any of the errors. In this situation some data may be lost from the restore but we *should* be able to restore the individual files as long as they're not system files. It's going to be a long long long night. Quote Link to comment Share on other sites More sharing options...
ooxy Posted June 21, 2011 Report Share Posted June 21, 2011 Hey, Thanks for the updates, I understand the pains of restoring from backups, finding myself doing it quite often because the sys admins at work are incapable of finding the on button for some things most of the time. Best of luck with it and I hope you don't have to go the manual approach. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 If you have a cPanel backup of your account ( a full backup ) open a support ticket and we can restore that backup to another server to get you back online quicker. Beyond that, we're doing what we can to restore the service as quickly as possible. Quote Link to comment Share on other sites More sharing options...
KevinD872 Posted June 21, 2011 Report Share Posted June 21, 2011 I'm sure you are frustrated and it sounds like you're having a bad day overall, Michael. I appreciate your efforts too, but I think what techshots was saying was to have some kind of backups that can be run on a new server while the old one is being fixed, etc. The problem I see with this is that I've no doubt that your backups are compressed and in an unusable state as-is. It's not like you could just pop a backup HDD out of a server, throw it in a new one and off you go (AFAIK). But if this could be done somehow........ I don't claim to be any kind of expert on web hosting, so I am not sure how things could be done differently on your end to prevent this kind of down-time, but I do hope that you are considering some kind of new plan for the future. I know you can't prevent DDoS attacks, but perhaps some other method of backups that would allow them to be almost instantly put into another server and back up and running within an hour. Is there a way to offer bare-bones service (such as a "technical difficulties" home page and email services using each domain's email account settings as of a last back-up configuration)? Again, I don't know much about web hosting & servers. I'm just trying to throw out some ideas. While I'm undoubtedly not suffering nearly as badly as some of your other customers on Fresco (my website isn't my sole source of income or anything like that), losing my website and especially email for 24 hours (which is about how long it will be now assuming your next backup works) is not good. I do realize this has come about after an unfortunate series of events and otherwise you usually offer top-notch service. You can feel a little better knowing that you still have days to go in order to beat the down-time record of my old host after they botched a server migration, but I hope you don't even come close. Good luck with the next rescue attempt.... hopefully we'll all have some better news by the morning. Quote Link to comment Share on other sites More sharing options...
jack Posted June 21, 2011 Report Share Posted June 21, 2011 Does anyone have any experience with something such as WHMautobackup? While it's useless now something like this would allow me to restore accounts on my VPS in the future. Quote Link to comment Share on other sites More sharing options...
FloridaClay Posted June 21, 2011 Report Share Posted June 21, 2011 I'm not an expert, who can explain this to me?: In this situation some data may be lost from the restore but we *should* be able to restore the individual files as long as they're not system files. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 I'm sure you are frustrated and it sounds like you're having a bad day overall, Michael. I appreciate your efforts too, but I think what techshots was saying was to have some kind of backups that can be run on a new server while the old one is being fixed, etc. The problem I see with this is that I've no doubt that your backups are compressed and in an unusable state as-is. It's not like you could just pop a backup HDD out of a server, throw it in a new one and off you go (AFAIK). But if this could be done somehow........ If it were that simple, we'd have already done that and had everybody back online. Our goal is to keep everybody online as much as possible and to restore service as quickly as possible when things do go offline. Unfortunately we can't have double the number of servers online at all times so that we can have a backup when something goes wrong - to offer this would take the price of plans from $7.50~75.50/mo to the range of $50~500/mo range. I don't claim to be any kind of expert on web hosting, so I am not sure how things could be done differently on your end to prevent this kind of down-time, but I do hope that you are considering some kind of new plan for the future. I know you can't prevent DDoS attacks, but perhaps some other method of backups that would allow them to be almost instantly put into another server and back up and running within an hour.Short of buying duplicate hardware to make this happen, which we can't do, there is no better way. Beyond that - duplicating the entire contents of our high end shared hosting servers daily in a fashion that would allow us to simply pop in those drives or switch servers would be so intensive that the servers would slow to a crawl during the backup process. Is there a way to offer bare-bones service (such as a "technical difficulties" home page and email services using each domain's email account settings as of a last back-up configuration)? Again, I don't know much about web hosting & servers. I'm just trying to throw out some ideas.It's certainly possible for you to do, if you wish for your domains, but there's no way for us to do that for you. It would take us days if not weeks to do this for every domain on the server and this is time better spent restoring the service. While I'm undoubtedly not suffering nearly as badly as some of your other customers on Fresco (my website isn't my sole source of income or anything like that), losing my website and especially email for 24 hours (which is about how long it will be now assuming your next backup works) is not good. I do realize this has come about after an unfortunate series of events and otherwise you usually offer top-notch service. You can feel a little better knowing that you still have days to go in order to beat the down-time record of my old host after they botched a server migration, but I hope you don't even come close. If email is critical, I would suggest using something like Google Apps Good luck with the next rescue attempt.... hopefully we'll all have some better news by the morning.I'm hoping it goes well as well. On a side-note we did reconfigure the raid array a little differently to increase responsiveness and performance on the server once it is online and operational. I wasn't going to announce it but I figure I might a well at least let us know we're all going to get something improved out of this, not that it makes up for it. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Does anyone have any experience with something such as WHMautobackup? While it's useless now something like this would allow me to restore accounts on my VPS in the future.I've never used it but some of our customers do. Do please open a new thread to discuss this, however. I'm not an expert, who can explain this to me?: In this situation some data may be lost from the restore but we *should* be able to restore the individual files as long as they're not system files.What it means is that if we can get the server to boot up and come online, we should still be able to pull a good copy of that file out of the backup system from some point in time. If the system files are damaged (i.e. server will not boot) then we're going to have to go the hard route and do the restoration entirely manually. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Once we finish this restoration from an additional day back, should the file system still have issues we're going to go ahead and take the time to run a file system check again and see if we can't get it to boot. If not, we'll go to the next step. The ETA on this restoration is about the same as the last, approximately 7 hours from start to finish. I will update this thread periodically. Quote Link to comment Share on other sites More sharing options...
Vilandra Posted June 21, 2011 Report Share Posted June 21, 2011 Just want to say thank you to Mike (and team) for being on top of this - in the years I spent with my last host they never ever posted in their "Known Issues" forum about the server I was on, so it's nice to actually be getting updates. Quote Link to comment Share on other sites More sharing options...
jack Posted June 21, 2011 Report Share Posted June 21, 2011 Forgive me if this is a stupid question but is there no way to retrieve backups from the backup server? I'm looking at loosing a far few customers without a doubt. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Forgive me if this is a stupid question but is there no way to retrieve backups from the backup server? I'm looking at loosing a far few customers without a doubt.While we're conducting a restoration from the server, any other restoration requests would be queued by the backup server. Beyond that, the data isn't stored in the backup server in a fashion that would be easily restored to another account. For example, even if you were able to issue a restoration request - you would only get files and no databases. We'll have everything back online as quickly as possible. Quote Link to comment Share on other sites More sharing options...
SpecialOccasions Posted June 21, 2011 Report Share Posted June 21, 2011 Honestly guys this is really bad now. As your sleeping, we've started another work day in Dubai and still got nothing running! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Honestly guys this is really bad now. As your sleeping, we've started another work day in Dubai and still got nothing running!You're not the only one, and I wish I were sleeping. Quote Link to comment Share on other sites More sharing options...
jack Posted June 21, 2011 Report Share Posted June 21, 2011 It's 3:53 AM here in England. I've got clients moaning at me and they have every right because this is allot of downtime. I will definitely be providing 1 month free credit to all clients effected. The worse thing about this is sending an email saying 5 hours ETA and then having to wait 7 hours after the failed restoration. It's not possible to explain this to my clients. No doubt these things happen but it does seem that downtime shouldn't be this much. I'll definitely be looking into some more sfaety precautions. Quote Link to comment Share on other sites More sharing options...
SpecialOccasions Posted June 21, 2011 Report Share Posted June 21, 2011 Sp whats the ETA now? You're not the only one, and I wish I were sleeping. Quote Link to comment Share on other sites More sharing options...
Software Posted June 21, 2011 Report Share Posted June 21, 2011 Trust me. I'm in the USA and will not sleep either! This killed me today! Quote Link to comment Share on other sites More sharing options...
jack Posted June 21, 2011 Report Share Posted June 21, 2011 We just have to hope that this restoration works. If it doesn't I don't see anyway of keeping my clients. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 It's 3:53 AM here in England. I've got clients moaning at me and they have every right because this is allot of downtime. I will definitely be providing 1 month free credit to all clients effected.And I promise you that we have more clients moaning at us than you have moaning at you. I'm not trying to put you down or anything - but you do have to realize that we are in the same situation as you with clients being upset. Only a very small portion of the clients affected are posting here on the forums. The worse thing about this is sending an email saying 5 hours ETA and then having to wait 7 hours after the failed restoration. It's not possible to explain this to my clients.That's exactly why it's an estimation. Which would you prefer? Us saying "It will be back when it's back." or us attempting to give you an ETA? I do apologize that the original ETA was not met. We're obviously not intentionally keeping things offline - the best thing we can do is get everything back online as quickly as possible. I will tell you I've not slept in going on 48 hours as I was just about to go to sleep when these issues started happening yesterday. No doubt these things happen but it does seem that downtime shouldn't be this much. I'll definitely be looking into some more sfaety precautions.It's a good idea to keep your own backups either way (whether you're online 100% for 10 years or offline more than online). If you have your own backups of your accounts we'll be happy to restore them to an alternate server. Sp whats the ETA now?It's the same amount of time as the last restoration process, around 6 hours estimated from the start. I am editing the very first post of this thread with updates as well. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 We just have to hope that this restoration works. If it doesn't I don't see anyway of keeping my clients.For all of those asking for an ETA - keep in mind that if we have the same file system errors after this restoration we're simply going to have to do our best to repair the file system. This will take an additional 2 to 6 hours after the restoration, so we're crossing our fingers we don't have to do that. At the end of the day, the short version is that we'll have it all back online as soon as we possibly can. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.