Jump to content


Photo

[Completed] Fresco Server - Restoration Completed - File System Check Competed


  • Please log in to reply
218 replies to this topic

#81 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 08:22 PM

The file system is still reporting errors, we're going to try going back one additional day and cross our fingers that the file system is clean. If not, we'll simply need to run a file system check again after the restoration and get to work on fixing any of the errors. In this situation some data may be lost from the restore but we *should* be able to restore the individual files as long as they're not system files.

It's going to be a long long long night.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#82 ooxy

ooxy

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 20 June 2011 - 08:24 PM

Hey,

Thanks for the updates, I understand the pains of restoring from backups, finding myself doing it quite often because the sys admins at work are incapable of finding the on button for some things most of the time.

Best of luck with it and I hope you don't have to go the manual approach.
  • 0

#83 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 08:25 PM

If you have a cPanel backup of your account ( a full backup ) open a support ticket and we can restore that backup to another server to get you back online quicker. Beyond that, we're doing what we can to restore the service as quickly as possible.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#84 KevinD872

KevinD872

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:NW Indiana
  • Interests:Drums & music, motorcycles, computers, programming and gadgets.

Posted 20 June 2011 - 08:35 PM

I'm sure you are frustrated and it sounds like you're having a bad day overall, Michael. I appreciate your efforts too, but I think what techshots was saying was to have some kind of backups that can be run on a new server while the old one is being fixed, etc. The problem I see with this is that I've no doubt that your backups are compressed and in an unusable state as-is. It's not like you could just pop a backup HDD out of a server, throw it in a new one and off you go (AFAIK). But if this could be done somehow........ ;)

I don't claim to be any kind of expert on web hosting, so I am not sure how things could be done differently on your end to prevent this kind of down-time, but I do hope that you are considering some kind of new plan for the future. I know you can't prevent DDoS attacks, but perhaps some other method of backups that would allow them to be almost instantly put into another server and back up and running within an hour. Is there a way to offer bare-bones service (such as a "technical difficulties" home page and email services using each domain's email account settings as of a last back-up configuration)? Again, I don't know much about web hosting & servers. I'm just trying to throw out some ideas.

While I'm undoubtedly not suffering nearly as badly as some of your other customers on Fresco (my website isn't my sole source of income or anything like that), losing my website and especially email for 24 hours (which is about how long it will be now assuming your next backup works) is not good. I do realize this has come about after an unfortunate series of events and otherwise you usually offer top-notch service. You can feel a little better knowing that you still have days to go in order to beat the down-time record of my old host after they botched a server migration, but I hope you don't even come close. :D

Good luck with the next rescue attempt.... hopefully we'll all have some better news by the morning.
  • 0

#85 jack

jack

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 20 June 2011 - 08:40 PM

Does anyone have any experience with something such as WHMautobackup? While it's useless now something like this would allow me to restore accounts on my VPS in the future.
  • 0

#86 FloridaClay

FloridaClay

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 20 June 2011 - 08:41 PM

I'm not an expert, who can explain this to me?: In this situation some data may be lost from the restore but we *should* be able to restore the individual files as long as they're not system files.
  • 0

#87 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 08:43 PM

I'm sure you are frustrated and it sounds like you're having a bad day overall, Michael. I appreciate your efforts too, but I think what techshots was saying was to have some kind of backups that can be run on a new server while the old one is being fixed, etc. The problem I see with this is that I've no doubt that your backups are compressed and in an unusable state as-is. It's not like you could just pop a backup HDD out of a server, throw it in a new one and off you go (AFAIK). But if this could be done somehow........ ;)

If it were that simple, we'd have already done that and had everybody back online. Our goal is to keep everybody online as much as possible and to restore service as quickly as possible when things do go offline.

Unfortunately we can't have double the number of servers online at all times so that we can have a backup when something goes wrong - to offer this would take the price of plans from $7.50~75.50/mo to the range of $50~500/mo range.

I don't claim to be any kind of expert on web hosting, so I am not sure how things could be done differently on your end to prevent this kind of down-time, but I do hope that you are considering some kind of new plan for the future. I know you can't prevent DDoS attacks, but perhaps some other method of backups that would allow them to be almost instantly put into another server and back up and running within an hour.

Short of buying duplicate hardware to make this happen, which we can't do, there is no better way. Beyond that - duplicating the entire contents of our high end shared hosting servers daily in a fashion that would allow us to simply pop in those drives or switch servers would be so intensive that the servers would slow to a crawl during the backup process.

Is there a way to offer bare-bones service (such as a "technical difficulties" home page and email services using each domain's email account settings as of a last back-up configuration)? Again, I don't know much about web hosting & servers. I'm just trying to throw out some ideas.

It's certainly possible for you to do, if you wish for your domains, but there's no way for us to do that for you. It would take us days if not weeks to do this for every domain on the server and this is time better spent restoring the service.

While I'm undoubtedly not suffering nearly as badly as some of your other customers on Fresco (my website isn't my sole source of income or anything like that), losing my website and especially email for 24 hours (which is about how long it will be now assuming your next backup works) is not good. I do realize this has come about after an unfortunate series of events and otherwise you usually offer top-notch service. You can feel a little better knowing that you still have days to go in order to beat the down-time record of my old host after they botched a server migration, but I hope you don't even come close. :D

If email is critical, I would suggest using something like Google Apps :)

Good luck with the next rescue attempt.... hopefully we'll all have some better news by the morning.

I'm hoping it goes well as well. On a side-note we did reconfigure the raid array a little differently to increase responsiveness and performance on the server once it is online and operational. I wasn't going to announce it but I figure I might a well at least let us know we're all going to get something improved out of this, not that it makes up for it.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#88 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 08:45 PM

Does anyone have any experience with something such as WHMautobackup? While it's useless now something like this would allow me to restore accounts on my VPS in the future.

I've never used it but some of our customers do. Do please open a new thread to discuss this, however.

I'm not an expert, who can explain this to me?: In this situation some data may be lost from the restore but we *should* be able to restore the individual files as long as they're not system files.

What it means is that if we can get the server to boot up and come online, we should still be able to pull a good copy of that file out of the backup system from some point in time. If the system files are damaged (i.e. server will not boot) then we're going to have to go the hard route and do the restoration entirely manually.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#89 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 08:47 PM

Once we finish this restoration from an additional day back, should the file system still have issues we're going to go ahead and take the time to run a file system check again and see if we can't get it to boot. If not, we'll go to the next step.

The ETA on this restoration is about the same as the last, approximately 7 hours from start to finish. I will update this thread periodically.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#90 Vilandra

Vilandra

    Newbie

  • Members
  • Pip
  • 23 posts
  • Gender:Female
  • Location:Pittsburgh, PA
  • Interests:Chelsea FC!

Posted 20 June 2011 - 09:01 PM

Just want to say thank you to Mike (and team) for being on top of this - in the years I spent with my last host they never ever posted in their "Known Issues" forum about the server I was on, so it's nice to actually be getting updates. :)
  • 0

#91 jack

jack

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 20 June 2011 - 09:04 PM

Forgive me if this is a stupid question but is there no way to retrieve backups from the backup server?

I'm looking at loosing a far few customers without a doubt.
  • 0

#92 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 09:07 PM

Forgive me if this is a stupid question but is there no way to retrieve backups from the backup server?

I'm looking at loosing a far few customers without a doubt.

While we're conducting a restoration from the server, any other restoration requests would be queued by the backup server. Beyond that, the data isn't stored in the backup server in a fashion that would be easily restored to another account. For example, even if you were able to issue a restoration request - you would only get files and no databases.

We'll have everything back online as quickly as possible.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#93 SpecialOccasions

SpecialOccasions

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 20 June 2011 - 09:50 PM

Honestly guys this is really bad now. As your sleeping, we've started another work day in Dubai and still got nothing running!
  • 0

#94 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 09:51 PM

Honestly guys this is really bad now. As your sleeping, we've started another work day in Dubai and still got nothing running!

You're not the only one, and I wish I were sleeping.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#95 jack

jack

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 20 June 2011 - 09:53 PM

It's 3:53 AM here in England. I've got clients moaning at me and they have every right because this is allot of downtime. I will definitely be providing 1 month free credit to all clients effected.

The worse thing about this is sending an email saying 5 hours ETA and then having to wait 7 hours after the failed restoration. It's not possible to explain this to my clients.

No doubt these things happen but it does seem that downtime shouldn't be this much. I'll definitely be looking into some more sfaety precautions.
  • 0

#96 SpecialOccasions

SpecialOccasions

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 20 June 2011 - 09:56 PM

Sp whats the ETA now?

You're not the only one, and I wish I were sleeping.


  • 0

#97 Software

Software

    Newbie

  • Members
  • Pip
  • 16 posts

Posted 20 June 2011 - 09:57 PM

Trust me. I'm in the USA and will not sleep either! This killed me today!
  • 0

#98 jack

jack

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 20 June 2011 - 09:57 PM

We just have to hope that this restoration works. If it doesn't I don't see anyway of keeping my clients.
  • 0

#99 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 10:00 PM

It's 3:53 AM here in England. I've got clients moaning at me and they have every right because this is allot of downtime. I will definitely be providing 1 month free credit to all clients effected.

And I promise you that we have more clients moaning at us than you have moaning at you. I'm not trying to put you down or anything - but you do have to realize that we are in the same situation as you with clients being upset. Only a very small portion of the clients affected are posting here on the forums.

The worse thing about this is sending an email saying 5 hours ETA and then having to wait 7 hours after the failed restoration. It's not possible to explain this to my clients.

That's exactly why it's an estimation. Which would you prefer? Us saying "It will be back when it's back." or us attempting to give you an ETA? I do apologize that the original ETA was not met. We're obviously not intentionally keeping things offline - the best thing we can do is get everything back online as quickly as possible.

I will tell you I've not slept in going on 48 hours as I was just about to go to sleep when these issues started happening yesterday.

No doubt these things happen but it does seem that downtime shouldn't be this much. I'll definitely be looking into some more sfaety precautions.

It's a good idea to keep your own backups either way (whether you're online 100% for 10 years or offline more than online). If you have your own backups of your accounts we'll be happy to restore them to an alternate server.


Sp whats the ETA now?

It's the same amount of time as the last restoration process, around 6 hours estimated from the start. I am editing the very first post of this thread with updates as well.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#100 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,902 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 10:05 PM

We just have to hope that this restoration works. If it doesn't I don't see anyway of keeping my clients.

For all of those asking for an ETA - keep in mind that if we have the same file system errors after this restoration we're simply going to have to do our best to repair the file system. This will take an additional 2 to 6 hours after the restoration, so we're crossing our fingers we don't have to do that.

At the end of the day, the short version is that we'll have it all back online as soon as we possibly can.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users