panchulo Posted June 20, 2011 Report Share Posted June 20, 2011 this is bad... seriously Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 The file system check left the server in an inoperable condition and, as such, we're going to be performing a full restoration of the server back to around 1:30 AM GMT-5. We'll update this thread as soon as the restoration ETA is known which should be within 30 minutes. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 The restore is in progress and the ETA provided by the system is right at 7 hours. This can increase or decrease as it runs so keep in mind it's an estimated time of completion. http://www.screen-shot.net/2011-06-20_1416_234234.png Quote Link to comment Share on other sites More sharing options...
Software Posted June 20, 2011 Report Share Posted June 20, 2011 My web guy said yesterday when he was working on it he got MySQL errors a few times. Quote Link to comment Share on other sites More sharing options...
Software Posted June 20, 2011 Report Share Posted June 20, 2011 (edited) The restore is in progress and the ETA provided by the system is right at 7 hours. This can increase or decrease as it runs so keep in mind it's an estimated time of completion. <removed image from quote due to redundancy> Are websites going to be restored during this time or not until this is all done? I can not afford to lose a day of business. Honestly, I just can't. Edited June 20, 2011 by MikeDVB Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 Are websites going to be restored during this time or not until this is all done? I can not afford to lose a day of business. Honestly, I just can't.It's a restoration of the entire server and all data (a fresh server install does not have 470 GB of data, more like 3 GB). Once this restoration is done everything should come back online without issue. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 The ETA seems to have stabilized around 5 hours 30 minutes. We'll update this thread periodically as the restoration runs. Quote Link to comment Share on other sites More sharing options...
benwiggy Posted June 20, 2011 Report Share Posted June 20, 2011 MDD have been fantastic over the past year for me....but today this is extreme! Its hurting business :( MDD have been brilliant for me also - and several hours of downtime isn't the end of the world as a result of such a big DDOS attack. What's really important is no loss of data and prompt restoration. For those who complain of a lack of updates I know Mike won't take a break until everything is up and running again - and I would rather he was working on restoring services than constantly posting updates. As someone who has experience with half a dozen hosts - I am happy in the knowledge that MDD continues to deliver the best service around - and when attacks like this happen (which they inevitably do) - they are promptly resolved. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 Current restoration status: 21.5 GB / 470 GB. Quote Link to comment Share on other sites More sharing options...
Leah2 Posted June 20, 2011 Report Share Posted June 20, 2011 I was wondering what steps can I as a customer take in the future to ensure I don't lose an entire day. Mainly I am concerned about email. If we cannot get or send email our customers move on (yes... they are that impatient) & take their project with them. Not having our website up also hurts our business immensely. I saw you mentioned name servers earlier in the thread. Would purchasing private name servers through you (or using a service like CloudFlare) keep our email going during a DDoS attack like today's and the unfortunate crashing of the server? If yes, how much is it? BTW - thanks for keeping everyone updated! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 I was wondering what steps can I as a customer take in the future to ensure I don't lose an entire day. Mainly I am concerned about email. If we cannot get or send email our customers move on (yes... they are that impatient) & take their project with them. Not having our website up also hurts our business immensely. Not having your website online hurts us immensely as well, as we have more than just a few offline unfortunately. If email is absolutely critical to your business I'd personally suggest using something like the free edition of Google Apps, although, I'm not sure about POP3/SMTP with that - I do believe you may have to have the paid version for that. I saw you mentioned name servers earlier in the thread. Would purchasing private name servers through you (or using a service like CloudFlare) keep our email going during a DDoS attack like today's and the unfortunate crashing of the server? If yes, how much is it?No, your email is hosted on-server. The nameserver message was specific to the specific client and really doesn't even belong in this thread as it's not exactly related. BTW - thanks for keeping everyone updated!Sure thing, I wish we didn't have to do this at all but that's unfortunately not something we have control over at this point. Current Progress: 25.7 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
Leah2 Posted June 20, 2011 Report Share Posted June 20, 2011 Not having your website online hurts us immensely as well, as we have more than just a few offline unfortunately. If email is absolutely critical to your business I'd personally suggest using something like the free edition of Google Apps, although, I'm not sure about POP3/SMTP with that - I do believe you may have to have the paid version for that. No, your email is hosted on-server. The nameserver message was specific to the specific client and really doesn't even belong in this thread as it's not exactly related. Sure thing, I wish we didn't have to do this at all but that's unfortunately not something we have control over at this point. Current Progress: 25.7 GB / 470 GB Thanks for the answer. It seemed too easy Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 Current Progress: 40 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
Themeltingclock Posted June 20, 2011 Report Share Posted June 20, 2011 Current Progress: 40 GB / 470 GBOuch. I was hoping to come back from lunch and find that our site was up and running. Looks like an entire day is shot at this point. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 Ouch. I was hoping to come back from lunch and find that our site was up and running. Looks like an entire day is shot at this point.It could be worse - we could be like a lot of providers out there and not have backups at all... Thankfully we're not like most. We're still investigating the original cause of the data corruption but I am not sure when we'll have those details. We are doing this restore to fresh, yet identical, hardware just in case it was hardware failure that caused the issue. Either way, I do apologize deeply to anybody who is experiencing this issue. Current progress: 53.4 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
Themeltingclock Posted June 20, 2011 Report Share Posted June 20, 2011 It could be worse - we could be like a lot of providers out there and not have backups at all... Thankfully we're not like most. We're still investigating the original cause of the data corruption but I am not sure when we'll have those details. We are doing this restore to fresh, yet identical, hardware just in case it was hardware failure that caused the issue. Either way, I do apologize deeply to anybody who is experiencing this issue. Current progress: 53.4 GB / 470 GBI'm thankful for your backups, no doubt. Do you have any idea of the target of the DDoS? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 I'm thankful for your backups, no doubt. Do you have any idea of the target of the DDoS?Unfortunately, no. We were working to ascertain that when the server went offline due to the file system issues. From the looks of it, however, the attack subsided a few hours ago. We're still able to see network traffic targeting the IPs on the server although the server is not serving web content at this time. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 Current Progress: 62 GB / 470 GB Quote Link to comment Share on other sites More sharing options...
Software Posted June 20, 2011 Report Share Posted June 20, 2011 So all of our websites will be restored to a certain point? Sorry, I am not the webmaster of my site, but rather the guy who loses money due to it being down. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 To 12:30 AM GMT-5 (EST) Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 Current progress: 75.5 GB of 470 GB restored. Quote Link to comment Share on other sites More sharing options...
Software Posted June 20, 2011 Report Share Posted June 20, 2011 Are their any precautions I can do on my end to avoid this next time? Quote Link to comment Share on other sites More sharing options...
Themeltingclock Posted June 20, 2011 Report Share Posted June 20, 2011 Are their any precautions I can do on my end to avoid this next time?Downside of shared hosting, homie. You're guilty by association of server resources. Correct me if I'm wrong, but if someone is hosted on your box that another person doesn't like and is the target of a DDoS, you're going down, too. Quote Link to comment Share on other sites More sharing options...
Themeltingclock Posted June 20, 2011 Report Share Posted June 20, 2011 Mike, I'm going back and reading this forum for my own knowledge - it looks like this is the 3rd(?) time Fresco has been DDoS'd in the last month. Any change of plans for the future? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted June 20, 2011 Author Report Share Posted June 20, 2011 Mike, I'm going back and reading this forum for my own knowledge - it looks like this is the 3rd(?) time Fresco has been DDoS'd in the last month. Any change of plans for the future?Short of not putting customers on the server, there is no way to prevent ddos attacks. The root cause for the current downtime is however file system corruption which is why I changed the thread title and edited the original post with an update. Current progress is 160 gb of 470 gb. It's going slower than expected. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.