[panchulo]

May the Force be with you Mike....

[ender]

still no any updates?

[SpecialOccasions]

MDD have been fantastic over the past year for me....but today this is extreme! Its hurting business

Frogsplash, on 20 June 2011 - 11:47 AM, said:

Yeah, MDD Hosting has been pretty rock solid for me so far, been with them about 6 months. They've been responsive, and even when hit with DDoS attacks, they have managed to be back online within 30 minutes in most cases.

This outage, however, has been really awful, relatively speaking. I've been fielding calls and emails all morning from grouchy clients that couldn't care less what a DDoS attack is – they just know their site is down, and I'd better fix it.

I'm not going to play pile on – not going to make a stressful night/day even worse – but if we could at least get an update, that would be appreciated.

[Software]

SpecialOccasions, on 20 June 2011 - 12:21 PM, said:

MDD have been fantastic over the past year for me....but today this is extreme! Its hurting business

To say the least! No phone support doesn't help either. Any update would be nice right now.

[panchulo]

any news ?

[panchulo]

this is bad... seriously

[MikeDVB]

The file system check left the server in an inoperable condition and, as such, we're going to be performing a full restoration of the server back to around 1:30 AM GMT-5. We'll update this thread as soon as the restoration ETA is known which should be within 30 minutes.

[MikeDVB]

The restore is in progress and the ETA provided by the system is right at 7 hours. This can increase or decrease as it runs so keep in mind it's an estimated time of completion.

[Software]

My web guy said yesterday when he was working on it he got MySQL errors a few times.

[Software]

MikeDVB, on 20 June 2011 - 01:16 PM, said:

The restore is in progress and the ETA provided by the system is right at 7 hours. This can increase or decrease as it runs so keep in mind it's an estimated time of completion.

<removed image from quote due to redundancy>

Are websites going to be restored during this time or not until this is all done? I can not afford to lose a day of business. Honestly, I just can't.

This post has been edited by MikeDVB: 20 June 2011 - 01:23 PM

[MikeDVB]

Software, on 20 June 2011 - 01:19 PM, said:

Are websites going to be restored during this time or not until this is all done? I can not afford to lose a day of business. Honestly, I just can't.

It's a restoration of the entire server and all data (a fresh server install does not have 470 GB of data, more like 3 GB). Once this restoration is done everything should come back online without issue.

[MikeDVB]

The ETA seems to have stabilized around 5 hours 30 minutes. We'll update this thread periodically as the restoration runs.

[benwiggy]

SpecialOccasions, on 20 June 2011 - 12:21 PM, said:

MDD have been fantastic over the past year for me....but today this is extreme! Its hurting business

MDD have been brilliant for me also - and several hours of downtime isn't the end of the world as a result of such a big DDOS attack. What's really important is no loss of data and prompt restoration.

For those who complain of a lack of updates I know Mike won't take a break until everything is up and running again - and I would rather he was working on restoring services than constantly posting updates.

As someone who has experience with half a dozen hosts - I am happy in the knowledge that MDD continues to deliver the best service around - and when attacks like this happen (which they inevitably do) - they are promptly resolved.

[MikeDVB]

Current restoration status: 21.5 GB / 470 GB.

[Leah2]

I was wondering what steps can I as a customer take in the future to ensure I don't lose an entire day. Mainly I am concerned about email. If we cannot get or send email our customers move on (yes... they are that impatient) & take their project with them. Not having our website up also hurts our business immensely.

I saw you mentioned name servers earlier in the thread. Would purchasing private name servers through you (or using a service like CloudFlare) keep our email going during a DDoS attack like today's and the unfortunate crashing of the server? If yes, how much is it?

BTW - thanks for keeping everyone updated!

[MikeDVB]

Leah2, on 20 June 2011 - 01:35 PM, said:

I was wondering what steps can I as a customer take in the future to ensure I don't lose an entire day. Mainly I am concerned about email. If we cannot get or send email our customers move on (yes... they are that impatient) & take their project with them. Not having our website up also hurts our business immensely.

Not having your website online hurts us immensely as well, as we have more than just a few offline unfortunately. If email is absolutely critical to your business I'd personally suggest using something like the free edition of Google Apps, although, I'm not sure about POP3/SMTP with that - I do believe you may have to have the paid version for that.

Leah2, on 20 June 2011 - 01:35 PM, said:

I saw you mentioned name servers earlier in the thread. Would purchasing private name servers through you (or using a service like CloudFlare) keep our email going during a DDoS attack like today's and the unfortunate crashing of the server? If yes, how much is it?

No, your email is hosted on-server. The nameserver message was specific to the specific client and really doesn't even belong in this thread as it's not exactly related.

Leah2, on 20 June 2011 - 01:35 PM, said:

BTW - thanks for keeping everyone updated!

Sure thing, I wish we didn't have to do this at all but that's unfortunately not something we have control over at this point.

Current Progress: 25.7 GB / 470 GB

[Leah2]

MikeDVB, on 20 June 2011 - 01:38 PM, said:

Not having your website online hurts us immensely as well, as we have more than just a few offline unfortunately. If email is absolutely critical to your business I'd personally suggest using something like the free edition of Google Apps, although, I'm not sure about POP3/SMTP with that - I do believe you may have to have the paid version for that.

No, your email is hosted on-server. The nameserver message was specific to the specific client and really doesn't even belong in this thread as it's not exactly related.

Sure thing, I wish we didn't have to do this at all but that's unfortunately not something we have control over at this point.

Current Progress: 25.7 GB / 470 GB

Thanks for the answer. It seemed too easy

[MikeDVB]

Current Progress: 40 GB / 470 GB

[Themeltingclock]

MikeDVB, on 20 June 2011 - 01:47 PM, said:

Current Progress: 40 GB / 470 GB

Ouch. I was hoping to come back from lunch and find that our site was up and running. Looks like an entire day is shot at this point.

[MikeDVB]

Themeltingclock, on 20 June 2011 - 01:55 PM, said:

Ouch. I was hoping to come back from lunch and find that our site was up and running. Looks like an entire day is shot at this point.

It could be worse - we could be like a lot of providers out there and not have backups at all... Thankfully we're not like most. We're still investigating the original cause of the data corruption but I am not sure when we'll have those details. We are doing this restore to fresh, yet identical, hardware just in case it was hardware failure that caused the issue.

Either way, I do apologize deeply to anybody who is experiencing this issue.

Current progress: 53.4 GB / 470 GB