100 replies to this topic

[MikeDVB]

We're going to be performing an upgrade on our backup system once this restoration is completed to speed up any future restorations. Hopefully we won't ever need to use it at this level again but at least the upgrade will help should the case ever arise.

[kocchi]

Yikes, so that's why everything's down.
Thanks for the update, Mike!

[kuemerle5]

*Sigh* A sad, sad day for MDD customers. It's good to know how hard Mike works to get these things fixed and is straight up honest with his clients. Also, if anyone want's specific details on the exploit, I'm pretty sure it's one of the following:

• [Redacted for security]
• [Redacted for security]

But by all means, I am no expert at this stuff so take the exploit guesses with a grain of salt. Thankfully, this isn't Go Daddy or 1 & 1. God knows what kind of PR garbage they would spit out and how long it would take them to actually fix the situation (I can assure you, it would probably a lot more than 2 days).

Also, if this ever happens again, could you tweet for clients to check the forums as I do not frequent them everyday? Thanks a lot Mike!

[MikeDVB]

*Sigh* A sad, sad day for MDD customers. It's good to know how hard Mike works to get these things fixed and is straight up honest with his clients. Also, if anyone want's specific details on the exploit, I'm pretty sure it's one of the following:

• [Redacted for security]
• [Redacted for security]

But by all means, I am no expert at this stuff so take the exploit guesses with a grain of salt. Thankfully, this isn't Go Daddy or 1 & 1. God knows what kind of PR garbage they would spit out and how long it would take them to actually fix the situation (I can assure you, it would probably a lot more than 2 days).

Official Operating System bug report and exploit details:
https://bugzilla.red...d=CVE-2010-3081
https://access.redha.../docs/DOC-40265

We waited to post this information to make sure that our other services were secured against this exploit however these links will provide the technical details as to what has happened for any that may be interested. The last thing we wanted to do is post details on the exploit for another server to be compromised by the same exploit before it could be mitigated.

I did edit out your links as they linked directly to the code used to perpetrate the attack and as such we'd like to avoid disbursing exploitation code.

Also, if this ever happens again, could you tweet for clients to check the forums as I do not frequent them everyday? Thanks a lot Mike!

My best piece of advice is to subscribe to this forum section, we always make a thread if there is an issue. To be honest I should have tweeted but I was focused on identifying the attack, securing the other servers against the attack, and restoring the server that was compromised.

[Dan S]

My best piece of advice is to subscribe to this forum section, we always make a thread if there is an issue.

Assuming that update email actually finds its way into your Inbox. One would have to use an email address not attached to the server(s) having issues.

[MikeDVB]

Assuming that update email actually finds its way into your Inbox. One would have to use an email address not attached to the server(s) having issues.

Which is advised anyways - you generally do not want to use an email account hosted by a provider as a support contact for that provider no matter who the provider is.

[karrinina]

Just wanted to comment in support of the open and responsive way this is being handled. This is the ONLY time I've observed unscheduled downtime in over a year of using the service, and in contrast to the extremely poor way a similar situation was handled by another one of our hosting providers a few weeks ago - this is a breath of fresh air!

[patlaw]

Which is advised anyways - you generally do not want to use an email account hosted by a provider as a support contact for that provider no matter who the provider is.

With Gmail accounts being free and easily accessible by POP and IMAP, it makes sense to have a spare email account for such purposes. Create an account, configure it for POP access on your main email client, and use it as the contact email for your server contact. That's what I do with pobox.com, which is an email aliasing services. When my echo POP box died, I pointed pobox and my email client to a different POP box, so at least that account is operating normally.

[MikeDVB]

Just wanted to comment in support of the open and responsive way this is being handled. This is the ONLY time I've observed unscheduled downtime in over a year of using the service, and in contrast to the extremely poor way a similar situation was handled by another one of our hosting providers a few weeks ago - this is a breath of fresh air!

Often times what is more important than what happens, is how it is handled. It's best to let our clients know exactly what is happening, why it happened, and what is being done about it. We've planned some major changes for our backup system moving forward from this issue so should we ever have to do anything like this again the restore should be magnitudes faster.

The restoration process is still in progress, I'll update the thread if anything changes.

[patlaw]

Please start including an ETA, if possible. One of my hosting clients is an attorney who is totally non-technical. It takes an hour to walk her through setup changes in her email client. I'd rather not have to go through the process of moving her to a new server. She has to use the echo IP address because of problems she was having with her ISP.

[MikeDVB]

Please start including an ETA, if possible. One of my hosting clients is an attorney who is totally non-technical. It takes an hour to walk her through setup changes in her email client. I'd rather not have to go through the process of moving her to a new server. She has to use the echo IP address because of problems she was having with her ISP.

We're looking at somewhere around 10 PM ~ Midnight EST (GMT-5) for the server to be online where we can begin restoring account data. Every time I've posted an ETA it is getting extended due to the backup system continually slowing down more and more.

None of the times posted in this thread are set in stone and are best educated guesses based upon the rate of transfer and the data left to transfer. Things could pick up significantly 1/2 way through the restore and finish way ahead of schedule or slow down even more and take substantially longer.

Right now what is being restored is the OS and MySQL databases as the MySQL databases are stored on a partition with system information required for booting. Once the server is online we'll begin restoring the partition which contains all of the hosting files (e.g. your /public_html and email accounts).

[MikeDVB]

The backup speed picked up quite a bit - if the speed continues the server itself will be online within an hour and we'll begin brining accounts online.

[MikeDVB]

Unfortunately doing the restore with the server online isn't going to be feasible, we're having to do a full restore of the server from start to finish before brining any accounts online. This process is going to take the full 24 to 48 hours and we'll update you if there are any changes.

[Brad]

Heya Michael. Great job you are doing and thank you for providing us with a great deal of feedback on the situation. At this point I'm wondering what to do in regards to informing my members and readers of what is going on. At the moment there is nothing that informs the visitors of the Echo websites about what is going on. For all our visitors know we all just packed up and left which leaves me worrying about my visitation numbers. 3 days down is a painfully long time.

Is there something you can do to inform our visitors of the situation? A basic notice page for all domain names associated with the Echo server would be a nice temporary means. Maybe a temp server could do this while Echo is being restored?

Otherwise, should we temp forward our domains or would it be pointless considering propagation?

Thank you!

[MikeDVB]

Heya Michael. Great job you are doing and thank you for providing us with a great deal of feedback on the situation. At this point I'm wondering what to do in regards to informing my members and readers of what is going on. At the moment there is nothing that informs the visitors of the Echo websites about what is going on. For all our visitors know we all just packed up and left which leaves me worrying about my visitation numbers. 3 days down is a painfully long time.

Is there something you can do to inform our visitors of the situation? A basic notice page for all domain names associated with the Echo server would be a nice temporary means. Maybe a temp server could do this while Echo is being restored?

Otherwise, should we temp forward our domains or would it be pointless considering propagation?

Thank you!

I've considered setting something like this up however to do so we'd have to edit every DNS zone on the server to point to another server and then when it came time to power Echo back on all of this work would have to manually be reversed which would result in even more downtime even once the server is back online.

If you do have a cPanel backup of your account we can restore it to another one of our servers and bring you online at least partially between now and when the restoration is completed.

[Dan S]

Unfortunately doing the restore with the server online isn't going to be feasible, we're having to do a full restore of the server from start to finish before brining any accounts online. This process is going to take the full 24 to 48 hours and we'll update you if there are any changes.

I have had my own domain since May, 1998 and I have never experienced a hosting outage of this magnitude. I work in IT and my boss' head would be on the platter if a server was down this long, whether it served internal or external customers. Where's the redundancy?

The open, honest communication is great, but...

[Brad]

Hmmm, ok. I think I have a full backup somewhere that we can make due with in the meantime. I know I have 5 day old database backups for sure. Should I ticket MDD?

[kocchi]

Otherwise, your next best bet is to sign up for a temporary host somewhere and change your DNS Host to them (or point to their servers, depending on if you could manage the DNS), and put up a message there. Make sure to keep your Time-to-Live as low as possible.

Just curious, whats the default TTL of domains on MDDHosting's DNS? Maybe they should be reduced for the time being in case anyone wants to point to a different server temporarily.

[Scott]

Hmmm, ok. I think I have a full backup somewhere that we can make due with in the meantime. I know I have 5 day old database backups for sure. Should I ticket MDD?

If you would like to use this backup to be restored to a new server, open a ticket and we can make that happen for you. If you would rather wait for the more up-to-date backup, then I would suggest waiting.

[MikeDVB]

I have had my own domain since May, 1998 and I have never experienced a hosting outage of this magnitude. I work in IT and my boss' head would be on the platter if a server was down this long, whether it served internal or external customers. Where's the redundancy?

If you want a load-balanced fail-over setup that wouldn't be affected by this type of issue you'd be paying $75+/month just for a shared account. The issue there is that it really depends on how the data is replicated across the balancer as to whether that would even help in an exploitation situation. It'd save your day in the event of hardware failure but in a system-level root kernel exploit it would be helpless to prevent the damage.

This was a zero-day kernel exploit - I'll be honest that we *probably* could have just restored the defaced/deleted data and gone from there and had much less downtime but my question is this: What happens if we take this shortcut and then the server is re-exploited through a hidden back-door and your data is not only lost but stolen?

It's not a risk we're willing to take - we're restoring the server back to a point in time before the attack happened to be sure that the server is secure and we're going to mitigate the exploit before bringing the public network online.

If you have your own backup of your account, open a ticket and we'll restore it to another server and get you back online very quickly. If you don't, you're going to have to wait for the server to be restored from the backup.

Our backup system is supposed to be able to do a full server restore in 5 to 10 hours however due to unforeseen circumstances it's taking substantially longer. We've done everything we can to speed up the process however there is only so much that can be done at this point.

If you do have any further questions you're welcome to post them.