100 replies to this topic

[Chy]

We're restoring the server back to a time before any files were modified and the system was compromised.

Thanks for the confirmation.

[Betacentauro]

Thanks for the answer Michael, i never has any to complain about the services and your help wen we are in some king of problems, but wen all this think is over we need to be sure this kind of brake downs don't happen again and i'm talking by my self i need some tips and a will need some help or explanation how to make a back up and restore the data in other secondary server. my business cant take one more of this. and many other are in the same situation, i'm not an expert in hosting and DNS stuff, is not my field in that one i trust MDD with eyes close. I hope we can get out of this problem as soon as possible and look forward for ways to prevent this king brake downs, thanks for all your HARD work and understanding.

[MikeDVB]

Thanks for the answer Michael, i never has any to complain about the services and your help wen we are in some king of problems, but wen all this think is over we need to be sure this kind of brake downs don't happen again and i'm talking by my self i need some tips and a will need some help or explanation how to make a back up and restore the data in other secondary server. my business cant take one more of this. and many other are in the same situation, i'm not an expert in hosting and DNS stuff, is not my field in that one i trust MDD with eyes close. I hope we can get out of this problem as soon as possible and look forward for ways to prevent this king brake downs, thanks for all your HARD work and understanding.

We'll be more than happy to help you with setting up your own backup plans and own contingency plans once this is all resolved. Just open a ticket once everything is back online.

[MikeDVB]

Ksplice has just sent us an email letting us know that they've pushed out a patch for the issue that caused the downtime. I'm quoting the email below:

Subject: [Ksplice][RHEL 5 Updates] New updates available via Ksplice (CVE-2010-3081)
Message:
Synopsis: CVE-2010-3081 can now be patched using Ksplice
CVEs: CVE-2010-3081

Systems running Red Hat Enterprise Linux 5 and CentOS 5 can now use
Ksplice to patch against CVE-2010-3081.

Ksplice is now providing an update for the high profile security
vulnerability CVE-2010-3010. Ksplice does not normally publish rebootless
updates for RHEL or CentOS before Red Hat has finished releasing a new
kernel, but in this case due to the high profile of this security
vulnerability, the fact that other distributions have successfully
provided this update, and our communications with the Red Hat security
team, we are now making this update available for customers to install.

Please note that the mitigation steps described at
<https://access.redha...docs/DOC-40265>, while effective against one
public exploit for CVE-2010-3081, do not actually correct this
vulnerability. A modified version of this exploit is effective even
against machines that have used the published Red Hat mitigation approach.
The only known effective solution to CVE-2010-3081 is to update the
kernel.

INSTALLING THE UPDATES

We recommend that all Ksplice Uptrack RHEL 5 and CentOS 5 users
install these updates. You can install these updates by running:

# uptrack-upgrade -y

DESCRIPTION

* CVE-2010-3081: Privilege escalation through stack underflow in compat.

A flaw was found in the 32-bit compatibility layer for 64-bit systems.
User-space memory was allocated insecurely when translating system
call inputs to 64-bit. A stack pointer underflow could occur when
using the "compat_alloc_user_space" method with an arbitrary length
input, as in getsockopt.

We've patched all servers against this vulnerability.

[mipra]

We've patched all servers against this vulnerability.

there you go....
Now, shall we all get some sleep?

[cvos]

If we would like to upload files to our account how may we do this? I can't access any site from a web browser, and the ping times out.

[Scott]

If we would like to upload files to our account how may we do this? I can't access any site from a web browser, and the ping times out.

Accessing your account is not possible until the server has been brought back online. Once service is restored, you will be able to access your account like normal.

[cvos]

when is it anticipated that the server will be up so we can update information? Also what is happening to email during this time?

[Scott]

when is it anticipated that the server will be up so we can update information? Also what is happening to email during this time?

Nothing has changed from what was stated above. The estimated time that things will be back is still about 2pm tomorrow, or twenty and a half hours from now.

As for email, most mail servers are configured to attempt to resend email for up to 72 hours, so most of your messages should come through.

[MikeDVB]

The Echo server is back online and operational ahead of schedule (by just under 12 hours). We do again sincerely apologize for this outage and appreciate your understanding as we did what was necessary to protect your account security and data integrity.

The server is going to be extra-busy for the next 24 to 48 hours as it gets slammed with mail that was sent to accounts on the server while the server was offline and other systems play catch-up. If you have any issues or questions feel free to post them here or to open a support ticket.

Thank you,

[kocchi]

Thanks!

[forumite]

I'd like to Thank Michael and the MDD Hosting team for the way in which they handled this situation. Yes, it was inconvenient, and probably tough for some folks, but Michael and Scott were open with us throughout the entire process. The pressure on these folks must have been tremendous, but they continued to respond to difficult questions in a calm, professional manner.

I have no doubt that Michael and his team will be working with the backup vendor to ensure the process, should it happen again in the future, will be smoother and faster.

[MikeDVB]

I'd like to Thank Michael and the MDD Hosting team for the way in which they handled this situation. Yes, it was inconvenient, and probably tough for some folks, but Michael aand Scot were open with us throughout the entire process. The pressure on these folks must have been tremendous, but Michael continued to respond to difficult questions in a calm, professional manner.

I have no doubt that Michael and his team will be working with the backup vendor to ensure the process, should it happen again in the future, will be smoother and faster.

We actually already have plans that are set into motion to ensure that any future restorations (large or small) complete much faster. We obviously hope to never have to use the system like this again but it is nice to know that should something happen that is outside our control (a system-level exploit or an act of god for example) we will be able to restore client data back either to the same hardware or to new hardware.

Being open and honest is company policy here at MDDHosting - many providers would have tried to cover the issue up or to spin it while we simply tell it like it is. I even went so far as to make sure that no posts in this thread were removed or censored in any way. We understand that our clients were (and likely still are) frustrated over this incident and if they wish to share their frustration, that's perfectly acceptable.

I've sent out an email to everyone advising some suggestions on keeping your own backups just in case as that's definitely a good step to take no matter where you are hosted, who your provider is (even if it's us), and what they promise.

If you have any questions, comments, or concerns about the outage, the restoration process, or anything else related to this issue by all means please feel free to let us know.

Edit: Clarification - I did actually "censor" one post but I didn't censor the spirit of the post, I just removed the direct links to the code used to exploit the server for security purposes.

Edited by MikeDVB, 19 September 2010 - 03:23 AM.
Clarification

[MjrNuT]

I second forumite's post!

:thumbsup: to MDD.

Look forward to just the ramp up now.

[Chy]

I don't think I've been this happy to wake up to notification alerts I had email. Thanks boys! You done good!

[Mike_M]

Thanks for all the hard work and keeping us informed along the way!!!

[Ivone]

Thanks Mike and MDD team. You did great job.

[frankacter]

While I'm not personally involved in this incident, it is very refreshing to see the transparent and professional way in which it was handled. Thanks to Mike and MDD team for their continued efforts, please do take time to refresh for both your health and sanity.

You mention you have "plans in motion".. something, as a client, I would appreciate seeing is a message to all clients in say 30 days that includes:

1) What steps have been put in place (actually executed on) as well as any related future steps to address the restore time.

2) A reminder to set up (and execute on schedule) a client level personal backup. A link the documented process / steps would be a nice touch.

3) Partnering with a 3rd party (or building yourself) for automated offsite client level backup service. I'd imagine this would be an optional service at a monthly fee. Something in the spirit of siteautobackup.com or backupalicious.com.

4) Automated server wide checks that can be run to audit and report when client apps/scripts/plugins are out of date. Thinking something like oldscriptfinder.com, but where the messaging is delivered directly to the client once a day/week/month depending on the critical nature of the out of date script. Perhaps offer incentive such as a monthly discount if a client's audits are clear of any outdated scripts :-)

[MikeDVB]

1) What steps have been put in place (actually executed on) as well as any related future steps to address the restore time.

Many of the technical details of what is being done most of our clients won't really care about I wouldn't think. I'll more than likely post them here on the forum and offer a link to those that are interested in further details.

2) A reminder to set up (and execute on schedule) a client level personal backup. A link the documented process / steps would be a nice touch.

It all really depends on how you want to go about doing things - whether you want to back up to your home computer, another web host, if you just want to backup your databases daily and your files weekly, etc... Ultimately it's the your responsibility to make sure that you have your own copy of your data. We always recommend that you run your own backups and we'll be more than happy to help anybody unsure how to do this on their own.

3) Partnering with a 3rd party (or building yourself) for automated offsite client level backup service. I'd imagine this would be an optional service at a monthly fee. Something in the spirit of siteautobackup.com or backupalicious.com.

Services such as these exist but the issue is that a vast majority of people choose their hosting based upon price and most aren't going to be willing to pay the additional fees to have their own backups. There are services out there that will automate your backups for you and there are services such as bqinternet.com where you can sign up for an FTP or RSYNC backup account and then you can use a simple script such as the one found here to back up your account to that external storage.

4) Automated server wide checks that can be run to audit and report when client apps/scripts/plugins are out of date. Thinking something like oldscriptfinder.com, but where the messaging is delivered directly to the client once a day/week/month depending on the critical nature of the out of date script. Perhaps offer incentive such as a monthly discount if a client's audits are clear of any outdated scripts :-)

That's an option we've considered in the past but ultimately the script security is the responsibility of the end-user. If every client on our servers is ok with us raising pricing by $1/month per account we could very easily add all of the above. Alternatively our clients could pay nothing additional to us and make sure to set up their own plans for backups and making sure their scripts are up to date.

There are always things that we could offer that we don't that some would see as a good idea. It's a balance at the provider's level to offer quality reliable service with high performance at a good price point.

From this situation - the only thing that really went wrong that we could have possibly had any control over was the speed at which the backup was restored. As I've said plans are being executed currently to ensure that any such restoration in the future will happen much faster.

I've seen many providers go through similar situations be it a system level exploit or hardware failure where data was lost and I can't think of any of them that were actually able to fully recover a copy of the server, in it's entirety. I'm not saying that it hasn't happened - but it's not common.

Now don't get me wrong - we are always evaluating new options and ways to expand our services and what we provide with our plans but not everything that sounds like a good idea would be feasibly implemented without raising prices.

We always do our best to keep our client base apprised of our improvements and changes that we make to the way we operate and features and services that we provide. Once we're done making changes to the way our backup systems operate we'll definitely let everybody know what was changed and what the benefits are.

[Shelley]

With regard to email - do you have any comments or ideas as to how incoming and outgoing were handled during the server downtime and restore. I have personally sent some as a test (from laptop to pc, different addresses) that have not been received. They were held in my outbox until the connection established during or after the restore and then sent. However not received.

I have a couple of clients who are anxious about emails that they should have received during this time - for one, an insurance agent, there is legal ramifications and therefore they need to have a bit more information as to how to address this. I am going to advise them to send out an eblast to their correspondents and ask them to resend all email from Thursday on, however I would like to be able to respond to their queries with some background.

thanks