jack Posted June 21, 2011 Report Share Posted June 21, 2011 I'm not blaming you Mike, this is clearing multiple problems all in the same time frame which is rare. You may have more clients moaning at you but you've got 10 servers, 9 of which I assume are online. Most of my hosting client base is on this single account, granted I should have had some sort of backup plan in place. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 I'm not blaming you Mike, this is clearing multiple problems all in the same time frame which is rare. You may have more clients moaning at you but you've got 10 servers, 9 of which I assume are online. Most of my hosting client base is on this single account, granted I should have had some sort of backup plan in place.We've been in your situation before believe it or not - we started on a reseller account with HostGator back in the day and moved to our own full dedicated server within the first 30 days due to downtime. Within that first month on the dedicated server (second month in business) the data center's power distribution room exploded (literally) and we were offline for a full 52 hours. Our site, our customers' sites, everything. People thought that we had up and disappeared as nobody had any way to get in contact with us. Life is hard, hosting is hard, you can't always predict what is going to happen and all that you can do is plan for the worst and hope that you've planned accordingly and that your plans work out well. If you want any advice on setting up your own backups of your accounts once everything is back online - open a ticket and ask for it to be escalated to me and I'll do my best to help/advise you. Quote Link to comment Share on other sites More sharing options...
Software Posted June 21, 2011 Report Share Posted June 21, 2011 Progress? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Progress?94.5 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
SpecialOccasions Posted June 21, 2011 Report Share Posted June 21, 2011 Apologies for the stupidity, but what does 6 hours from the start mean?How long approx from now, as im not sure when it started. And I promise you that we have more clients moaning at us than you have moaning at you. I'm not trying to put you down or anything - but you do have to realize that we are in the same situation as you with clients being upset. Only a very small portion of the clients affected are posting here on the forums. That's exactly why it's an estimation. Which would you prefer? Us saying "It will be back when it's back." or us attempting to give you an ETA? I do apologize that the original ETA was not met. We're obviously not intentionally keeping things offline - the best thing we can do is get everything back online as quickly as possible. I will tell you I've not slept in going on 48 hours as I was just about to go to sleep when these issues started happening yesterday. It's a good idea to keep your own backups either way (whether you're online 100% for 10 years or offline more than online). If you have your own backups of your accounts we'll be happy to restore them to an alternate server. It's the same amount of time as the last restoration process, around 6 hours estimated from the start. I am editing the very first post of this thread with updates as well. Quote Link to comment Share on other sites More sharing options...
Charles Posted June 21, 2011 Report Share Posted June 21, 2011 MDD Hosting has backups in cases like this!!They did this as part of their "business plan". They took the steps well before things went bad and not after. For those of you that own a business and are getting "income" from them, you has well should have things setupfor cases like this as well. Create your own "off-site" backups, failover DNS, etc... Issues like this can and DO happen all the time every single day. What matters is how the host handles things like this. Mike has posted SEVERAL updates as to what is going on along with the progress and that is way more than what most hosts do!! I am also on Fresco, granted it is not a business I do have weekly backups based on how often I update my site. If I were to update my site more often, I would have backups created more often. Also, all my backups are off-site. It would be stupid to created backups that stay on the same account that you just backed up. It would serve no use and is pointless. Mike, thanks for all that you have done and I hope things get resolved and you can get some sleep soon. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 @Charles - if you wish to open a ticket we can restore one of your backups (that you have) to another server to get you back online if you wish. Quote Link to comment Share on other sites More sharing options...
Charles Posted June 21, 2011 Report Share Posted June 21, 2011 @Charles - if you wish to open a ticket we can restore one of your backups (that you have) to another server to get you back online if you wish. @Mike, thanks for offering to do that but I will wait until Fresco comes back online. Quote Link to comment Share on other sites More sharing options...
Wittgenstein Posted June 21, 2011 Report Share Posted June 21, 2011 What a nightmare 5 my sites are offline. Hope this will get solved today. Thanks for the info Mike. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 What a nightmare 5 my sites are offline. Hope this will get solved today. Thanks for the info Mike.We'll get everything back online one way or another. I've just paid a rather large sum of money to another administrator who is super experienced with file system repairs/data restoration and recovery. If this restoration doesn't work then he's the best chance at getting everything back in a timely fashion. Quote Link to comment Share on other sites More sharing options...
HADNow Posted June 21, 2011 Report Share Posted June 21, 2011 Good that I have my emails on another server, so atleast I can communicate. I dont blame MDD because this could happen with any other server / provider. The good point is that MDD is communicating on a continuous basis which helps a bit. The bad part is customers down line dont understand such things. With sites down for over 24 hours, its hard to explain why such a thing occurs. Also hope and wish all sites are restored successfully. Wishing its sorted out at the earliest. Quote Link to comment Share on other sites More sharing options...
Themeltingclock Posted June 21, 2011 Report Share Posted June 21, 2011 For the future - I have my site making backups every 2 days and storing one on the server and sending on to the Google cloud. Of course, they are from the Backup and Migrate module in Drupal, so they only save the MySQL database and the Drupal settings. Is there a way to set up a cron to automate the backups from cPanel, as well? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Progress: 109.7 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 For the future - I have my site making backups every 2 days and storing one on the server and sending on to the Google cloud. Of course, they are from the Backup and Migrate module in Drupal, so they only save the MySQL database and the Drupal settings. Is there a way to set up a cron to automate the backups from cPanel, as well?There is, Google for "cPanel Backup Cron" Quote Link to comment Share on other sites More sharing options...
008Rohit Posted June 21, 2011 Report Share Posted June 21, 2011 Most of the DDoS attacks were on Fresco lately if I am not mistaken. I am on Gemini and it never had any problem till now since a File-System restoration on 29th of April. (Thankfully just a day before I moved my sites to it) Is there any specific domain on Fresco which is the target of the attackers? And I love the way you are constantly updating the status. Many big fishes don't do this keeping their clients in complete darkness. However, I was shocked to see that it takes so much time to restore 470GB of data. BTW, Is Fresco a shared hosting server like Gemini or a VPS or Semi-dedicated hosting server? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Fresco is shared, and no it's not any one particular domain or customer - each and every attack has been on a different IP. It's just bad luck - the internet is a hostile place. One thing people need to make sure they understand is that it's not us that's the target, but one of our customers (and a different one each time it's happened). With the right amount of money you can go to some of the more nefarious types of sites and actually order a DDoS attack on any site/domain you want. Somebody has something on their site you disagree with? They made you mad? Well - pay enough and you can take them offline. The biggest issue with DDoS attacks is the collateral damage (i.e. everybody else on the server affected). Honestly I believe there were file system issues on the server before the server needed restarted (otherwise the backup restoration would have came back up clean) and that it was only a matter of time before the server went read-only on it's own requiring a file system check or restoration. Quote Link to comment Share on other sites More sharing options...
Themeltingclock Posted June 21, 2011 Report Share Posted June 21, 2011 Fresco is shared, and no it's not any one particular domain or customer - each and every attack has been on a different IP. It's just bad luck - the internet is a hostile place. One thing people need to make sure they understand is that it's not us that's the target, but one of our customers (and a different one each time it's happened). With the right amount of money you can go to some of the more nefarious types of sites and actually order a DDoS attack on any site/domain you want. Somebody has something on their site you disagree with? They made you mad? Well - pay enough and you can take them offline. The biggest issue with DDoS attacks is the collateral damage (i.e. everybody else on the server affected). Honestly I believe there were file system issues on the server before the server needed restarted (otherwise the backup restoration would have came back up clean) and that it was only a matter of time before the server went read-only on it's own requiring a file system check or restoration. Why is it that response times to Gemini and Fresco are so vastly different? If they are both shared hosting environments and Pingdom is generating requests from the same location, how is it that Fresco takes a whole 200+ms longer? Sounds like Fresco has been a ticking time bomb judging by how many times it's mentioned just on the first page of the Server and Network Announcements forum and how long its been down in the last 24 hours (plus what you just said) Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Progress: 128.5 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 138.3 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
HADNow Posted June 21, 2011 Report Share Posted June 21, 2011 Why is it that response times to Gemini and Fresco are so vastly different? If they are both shared hosting environments and Pingdom is generating requests from the same location, how is it that Fresco takes a whole 200+ms longer? Sounds like Fresco has been a ticking time bomb judging by how many times it's mentioned just on the first page of the Server and Network Announcements forum and how long its been down in the last 24 hours (plus what you just said)I too agree with this point. I had also mentioned this in one of my support tickets as to why there is a huge variation in terms of ping speeds between all other servers and Fresco because fresco seems to take almost 2-5 times more compared to other servers of MDD. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Why is it that response times to Gemini and Fresco are so vastly different? If they are both shared hosting environments and Pingdom is generating requests from the same location, how is it that Fresco takes a whole 200+ms longer? Sounds like Fresco has been a ticking time bomb judging by how many times it's mentioned just on the first page of the Server and Network Announcements forum and how long its been down in the last 24 hours (plus what you just said)Fresco has a lot more heavy disk I/O accounts on it and since Disk I/O is one of the hardest things to monitor/limit it can be difficult. We did make a major change to the raid array for this last restoration that should alleviate this. Fresco has around 1/2 the number of accounts any other server has just due to how I/O heavy a lot of the accounts are. Beyond that, it doesn't make it any more a "ticking time bomb" than any other server. Hardware fails, file systems become corrupted, it's only a matter of time before it happens to any system. What is super important is having a backup and the abilities to repair the damage if needed. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 Current Progress: 182.5 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 189.1 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
Buschiq Posted June 21, 2011 Report Share Posted June 21, 2011 Just wanted to say Thank You for all the hard work you guys are putting into this. I can't imagine the stress levels you must be experiencing right now, take care of yourselves! Also, I guess that most of you affected already thought of this - but don't forget to use all your other sources to let your visitors know what's happening. Perhaps you have a Facebook group, twitter or something like that. We're using FB right now to communicate and answer all sorts of questions (not just down time related). Only a quick tip, it's easy to forget things when one is stressed out Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 21, 2011 Author Report Share Posted June 21, 2011 The restoration to the older point (i.e. one day back) appears to be taking longer than the restoration that was same-day. This is likely due to the way the CDP server works in storing only changed data every day. There is a good possibility we may still need to do a file system check after this restoration finishes. We will keep the thread updated, as we have been. Current Progress: 193.9 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.