Jump to content


Photo

[Completed] Fresco Server - Restoration Completed - File System Check Competed


  • Please log in to reply
218 replies to this topic

#101 jack

jack

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 20 June 2011 - 10:07 PM

I'm not blaming you Mike, this is clearing multiple problems all in the same time frame which is rare.

You may have more clients moaning at you but you've got 10 servers, 9 of which I assume are online. Most of my hosting client base is on this single account, granted I should have had some sort of backup plan in place.
  • 0

#102 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 10:10 PM

I'm not blaming you Mike, this is clearing multiple problems all in the same time frame which is rare.

You may have more clients moaning at you but you've got 10 servers, 9 of which I assume are online. Most of my hosting client base is on this single account, granted I should have had some sort of backup plan in place.

We've been in your situation before believe it or not - we started on a reseller account with HostGator back in the day and moved to our own full dedicated server within the first 30 days due to downtime. Within that first month on the dedicated server (second month in business) the data center's power distribution room exploded (literally) and we were offline for a full 52 hours. Our site, our customers' sites, everything. People thought that we had up and disappeared as nobody had any way to get in contact with us.

Life is hard, hosting is hard, you can't always predict what is going to happen and all that you can do is plan for the worst and hope that you've planned accordingly and that your plans work out well.

If you want any advice on setting up your own backups of your accounts once everything is back online - open a ticket and ask for it to be escalated to me and I'll do my best to help/advise you.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#103 Software

Software

    Newbie

  • Members
  • Pip
  • 16 posts

Posted 20 June 2011 - 10:26 PM

Progress?
  • 0

#104 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 10:43 PM

Progress?

94.5 GB / 470 GB
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#105 SpecialOccasions

SpecialOccasions

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 20 June 2011 - 10:50 PM

Apologies for the stupidity, but what does 6 hours from the start mean?
How long approx from now, as im not sure when it started.


And I promise you that we have more clients moaning at us than you have moaning at you. I'm not trying to put you down or anything - but you do have to realize that we are in the same situation as you with clients being upset. Only a very small portion of the clients affected are posting here on the forums.

That's exactly why it's an estimation. Which would you prefer? Us saying "It will be back when it's back." or us attempting to give you an ETA? I do apologize that the original ETA was not met. We're obviously not intentionally keeping things offline - the best thing we can do is get everything back online as quickly as possible.

I will tell you I've not slept in going on 48 hours as I was just about to go to sleep when these issues started happening yesterday.

It's a good idea to keep your own backups either way (whether you're online 100% for 10 years or offline more than online). If you have your own backups of your accounts we'll be happy to restore them to an alternate server.


It's the same amount of time as the last restoration process, around 6 hours estimated from the start. I am editing the very first post of this thread with updates as well.


  • 0

#106 Charles

Charles

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 20 June 2011 - 10:53 PM

MDD Hosting has backups in cases like this!!
They did this as part of their "business plan". They took the steps well before things went bad and not after.

For those of you that own a business and are getting "income" from them, you has well should have things setup
for cases like this as well. Create your own "off-site" backups, failover DNS, etc...

Issues like this can and DO happen all the time every single day.

What matters is how the host handles things like this. Mike has posted SEVERAL updates as to what is going on along with the progress and that is way more than what most hosts do!!


I am also on Fresco, granted it is not a business I do have weekly backups based on how often I update my site. If I were to update my site more often, I would have backups created more often.

Also, all my backups are off-site. It would be stupid to created backups that stay on the same account that you just backed up. It would serve no use and is pointless.

Mike, thanks for all that you have done and I hope things get resolved and you can get some sleep soon.
  • 0

#107 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 10:56 PM

@Charles - if you wish to open a ticket we can restore one of your backups (that you have) to another server to get you back online if you wish.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#108 Charles

Charles

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 20 June 2011 - 10:59 PM

@Charles - if you wish to open a ticket we can restore one of your backups (that you have) to another server to get you back online if you wish.


@Mike, thanks for offering to do that but I will wait until Fresco comes back online.
  • 0

#109 Wittgenstein

Wittgenstein

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 20 June 2011 - 11:03 PM

What a nightmare :( 5 my sites are offline. Hope this will get solved today. Thanks for the info Mike.
  • 0
Whereof one cannot speak, thereof one must be silent.

#110 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 11:10 PM

What a nightmare :( 5 my sites are offline. Hope this will get solved today. Thanks for the info Mike.

We'll get everything back online one way or another. I've just paid a rather large sum of money to another administrator who is super experienced with file system repairs/data restoration and recovery. If this restoration doesn't work then he's the best chance at getting everything back in a timely fashion.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#111 HADNow

HADNow

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 20 June 2011 - 11:11 PM

Good that I have my emails on another server, so atleast I can communicate.

I dont blame MDD because this could happen with any other server / provider.

The good point is that MDD is communicating on a continuous basis which helps a bit.

The bad part is customers down line dont understand such things. With sites down for over 24 hours, its hard to explain why such a thing occurs.

Also hope and wish all sites are restored successfully.

Wishing its sorted out at the earliest.
  • 0

#112 Themeltingclock

Themeltingclock

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 20 June 2011 - 11:15 PM

For the future - I have my site making backups every 2 days and storing one on the server and sending on to the Google cloud. Of course, they are from the Backup and Migrate module in Drupal, so they only save the MySQL database and the Drupal settings. Is there a way to set up a cron to automate the backups from cPanel, as well?
  • 0

#113 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 11:15 PM

Progress: 109.7 GB / 470 GB
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#114 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 11:15 PM

For the future - I have my site making backups every 2 days and storing one on the server and sending on to the Google cloud. Of course, they are from the Backup and Migrate module in Drupal, so they only save the MySQL database and the Drupal settings. Is there a way to set up a cron to automate the backups from cPanel, as well?

There is, Google for "cPanel Backup Cron"
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#115 008Rohit

008Rohit

    Member

  • Members
  • PipPip
  • 32 posts
  • Gender:Male
  • Location:Kolkata, India
  • Interests:Tech, SEO, Sports.

Posted 20 June 2011 - 11:16 PM

Most of the DDoS attacks were on Fresco lately if I am not mistaken. I am on Gemini and it never had any problem till now since a File-System restoration on 29th of April. (Thankfully just a day before I moved my sites to it)

Is there any specific domain on Fresco which is the target of the attackers? And I love the way you are constantly updating the status. Many big fishes don't do this keeping their clients in complete darkness. However, I was shocked to see that it takes so much time to restore 470GB of data.

BTW, Is Fresco a shared hosting server like Gemini or a VPS or Semi-dedicated hosting server?
  • 0

TechTage - Hosted on S1

Genuine Web Hosting Reviews - Read my MDDHosting Review


#116 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 11:21 PM

Fresco is shared, and no it's not any one particular domain or customer - each and every attack has been on a different IP. It's just bad luck - the internet is a hostile place. One thing people need to make sure they understand is that it's not us that's the target, but one of our customers (and a different one each time it's happened).

With the right amount of money you can go to some of the more nefarious types of sites and actually order a DDoS attack on any site/domain you want. Somebody has something on their site you disagree with? They made you mad? Well - pay enough and you can take them offline. The biggest issue with DDoS attacks is the collateral damage (i.e. everybody else on the server affected).

Honestly I believe there were file system issues on the server before the server needed restarted (otherwise the backup restoration would have came back up clean) and that it was only a matter of time before the server went read-only on it's own requiring a file system check or restoration.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#117 Themeltingclock

Themeltingclock

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 20 June 2011 - 11:38 PM

Fresco is shared, and no it's not any one particular domain or customer - each and every attack has been on a different IP. It's just bad luck - the internet is a hostile place. One thing people need to make sure they understand is that it's not us that's the target, but one of our customers (and a different one each time it's happened).

With the right amount of money you can go to some of the more nefarious types of sites and actually order a DDoS attack on any site/domain you want. Somebody has something on their site you disagree with? They made you mad? Well - pay enough and you can take them offline. The biggest issue with DDoS attacks is the collateral damage (i.e. everybody else on the server affected).

Honestly I believe there were file system issues on the server before the server needed restarted (otherwise the backup restoration would have came back up clean) and that it was only a matter of time before the server went read-only on it's own requiring a file system check or restoration.


Why is it that response times to Gemini and Fresco are so vastly different? If they are both shared hosting environments and Pingdom is generating requests from the same location, how is it that Fresco takes a whole 200+ms longer? Sounds like Fresco has been a ticking time bomb judging by how many times it's mentioned just on the first page of the Server and Network Announcements forum and how long its been down in the last 24 hours (plus what you just said)
  • 0

#118 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 11:39 PM

Progress: 128.5 GB / 470 GB
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#119 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 20 June 2011 - 11:48 PM

138.3 GB / 470 GB
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#120 HADNow

HADNow

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 20 June 2011 - 11:49 PM

Why is it that response times to Gemini and Fresco are so vastly different? If they are both shared hosting environments and Pingdom is generating requests from the same location, how is it that Fresco takes a whole 200+ms longer? Sounds like Fresco has been a ticking time bomb judging by how many times it's mentioned just on the first page of the Server and Network Announcements forum and how long its been down in the last 24 hours (plus what you just said)

I too agree with this point. I had also mentioned this in one of my support tickets as to why there is a huge variation in terms of ping speeds between all other servers and Fresco because fresco seems to take almost 2-5 times more compared to other servers of MDD.
  • 0




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users