Jump to content
MDDHosting Forums

[Resolved] Echo Server Repair


Recommended Posts

Notice:

Normal service has been restored. You may skip ahead to the post announcing this: http://forums.mddhosting.com/topic/319-resolved-echo-server-repair/page__view__findpost__p__1445

 

The remainder of this post is being left online for those who are interested. Once again, echo has been successfully restored and all services are back to normal.

 

------------------------

 

Hello,

 

Our Echo server was the target of zero-day kernel exploit. What this means in simple terms is that an attacker exploited a flaw in the operating system that is unknown to the operating system developers and as such there is not yet a patch for this issue.

 

Fortunately this is where our backup system comes into play as we're going to restore the server back 24 hours prior to the attack. The downside to this restoration is that it can easily take 10 to 20 hours for this process to complete. We've already started the restoration process so that we can get things back online as quickly as possible.

 

We have investigated the attack and determined that all the attacker has done is defaced index files (index.php, index.html, index.htm, etc...). We do not believe the attacker has stolen any data, dumped any databases, or modified any system files however we are going to do a full restore of the system just in case they left a back door into the system that we've not found.

 

We have taken some pro-active steps to help protect customers on other servers that are unaffected which includes disabling FTP and SSH for all accounts. For anybody on another server wishing to work on their sites we ask that you please use the "File Manager" from within your cPanel. We understand the frustration and trouble this may cause however it's better to have to take a couple of extra steps to work on your site than to face 10 to 20 hours of downtime or worse due to an unpatched exploit.

 

We always believe in being as open as possible about what is happening and what we're doing to resolve the issue. We do ask that if you have any questions about this issue to please post them in this thread so that we can centralize the disbursement of information to our client base.

 

We appreciate your understanding and patience in these trying times.

 

Update:

I'm posting this to the original post just to save new visitors to the thread from having to review the entire thread.

 

We're currently restoring the Operating System on the server back 24 hours prior to the exploit. Once the server OS is restored we're going to then begin restoring customer accounts from the backup system. This method will allow accounts to come online as they are restored instead of all accounts having to wait until the entire process is completed.

 

We're currently looking at approximately 48 hours for the last account to be restored however accounts will begin coming back online one at a time here within the next few hours.

 

We have no way of prioritizing which accounts are restored first or last and they will be restored as they are stored into the backup system. If you have any questions, do please read the rest of this thread and feel free to post your questions here.

Link to comment
Share on other sites

That's honest enough for me.

There is nothing appriciated then pure honesty. I wish you all the best with the patch :blink:

Hopefully it will be patched quickly, we'll update this thread and all outstanding support tickets if there are any changes. Right now we're waiting on the restoration on Echo to finish and for an official patch to be released.
Link to comment
Share on other sites

The restoration process is going a *lot* slower than we anticipated unfortunately. There is around 450 GB of data to restore and it looks as though it's going to take nearly 48 hours to restore unless the process speeds up somewhere along the way. Believe me - we don't want this to take any longer than it has to but I'd rather be upfront and honest about the time frame and it end up being much shorter than the other way around.

 

We do sincerely apologize for this downtime however we chose the route of keeping your data safe by restoring back to a time before the exploit to be sure that no malicious files/code was left on the server.

Link to comment
Share on other sites

The restoration process is going a *lot* slower than we anticipated unfortunately. There is around 450 GB of data to restore and it looks as though it's going to take nearly 48 hours to restore unless the process speeds up somewhere along the way. Believe me - we don't want this to take any longer than it has to but I'd rather be upfront and honest about the time frame and it end up being much shorter than the other way around.

 

We do sincerely apologize for this downtime however we chose the route of keeping your data safe by restoring back to a time before the exploit to be sure that no malicious files/code was left on the server.

FTP tried and upped as of my post.

Link to comment
Share on other sites

I certainly understand and appreciate the need for security; however, shutting down websites for 2 days is going to hurt a lot of people, especially people like myself who do a lot of business through the site. Is there anyway to temporarily restore these sites on functioning servers so there isn't 48 hours of downtime?
Link to comment
Share on other sites

I certainly understand and appreciate the need for security; however, shutting down websites for 2 days is going to hurt a lot of people, especially people like myself who do a lot of business through the site. Is there anyway to temporarily restore these sites on functioning servers so there isn't 48 hours of downtime?

 

I would like to now this as well. 48 hours with no access to my email and sites is not good at all.

 

MDD is top notch and I don't balme you for this at all but 48 hours without services is going to hurt alot of people.

Link to comment
Share on other sites

You mentioned that the FTP is re-enabled on all servers. I just tried logging into my account, but without much luck. Is my site part of this hack or is it something else?

 

If you are located on the echo server, then FTP would not be available as the server is still being restored. All other servers should have FTP back now. If your account is not on echo, please open a ticket at http://mddhosting.com/support so we may investigate for your account specifically.

 

Is email going to take 48 hours as well?

 

Unfortunately, yes. All services on echo, including email, are offline while the restore process is running.

Link to comment
Share on other sites

Thanks for the update MikeDVB.

 

Thanks for taking the precaution of removing access.

 

For clarification, you've noted a temporary mitigation has been determined. How is/was this tested?

 

Is the restore of Echo a bare metal level one?

 

Look forward to your reply when you've been able. I know you were up late....as I was. :blink:

 

Regards

Link to comment
Share on other sites

I certainly understand and appreciate the need for security; however, shutting down websites for 2 days is going to hurt a lot of people, especially people like myself who do a lot of business through the site. Is there anyway to temporarily restore these sites on functioning servers so there isn't 48 hours of downtime?

What is taking so long is the actual restoration of the data - whether we restore it back to the same hardware (which is what we are doing) or we restored it back to a different server - it's going to take up to 48 hours to complete.

 

We don't want you to be offline any longer than you have to.

 

 

I would like to now this as well. 48 hours with no access to my email and sites is not good at all.

 

MDD is top notch and I don't balme you for this at all but 48 hours without services is going to hurt alot of people.

Indeed it is, on a positive note - if you do have your own cPanel backup of your account (I always personally recommend keeping your own off-provider backups, even with us) we can restore that account to an alternate server in the meantime. Open a support ticket and we'll get to work on these on a first come first served basis.

 

Will the email I've been sent while the server is down filter through once it comes back online, or will it be lost business?

 

I appreciate attacks like this happen, but hopefully the impact is minimal.

In most cases mail servers will continue trying to send emails for up to 72 hours, there are some cases where they will not and I cannot promise that everything will get through to you.

 

 

Thanks for the update MikeDVB.

Thanks for taking the precaution of removing access.

For clarification, you've noted a temporary mitigation has been determined. How is/was this tested?

The temporary mitigation was provided by the software vendor (RedHat) and was tested on a development box. The script used to exploit access to the server was tested before and after the mitigation was put in place on the development server and worked beautifully.

 

Is the restore of Echo a bare metal level one?
Yes.
Link to comment
Share on other sites

I'm guessing that if I don't have my own Cpanel back up your SOL.

We have backups of your data, so you're not "SOL" in the sense that you could have been with another provider.

 

We're actually in contact with our backup vendor right now to see if we can get things restored any faster or if there is a better way to go about this. I'll see that this thread is updated if we change the action plan or anything else changes.

Link to comment
Share on other sites

After speaking with our backup software provider they have advised us to take a different backup restoration route.

 

Here is what is happening:

  1. We're restoring the operating system and system files
  2. Bringing the server online
  3. Restoring account files once the server is online.

 

The entire process is still going to take 48 hours however within about 5 hours from now sites will start to come back online. We have no way of choosing what accounts are restored first or last and it they will be restored in the order that they're stored in the backup system.

 

This will help prevent some additional downtime for a good portion of the clients affected and while we're still evaluating our options to get things back online more quickly this appears to be our best option at this point.

Link to comment
Share on other sites

*Sigh* A sad, sad day for MDD customers. It's good to know how hard Mike works to get these things fixed and is straight up honest with his clients. Also, if anyone want's specific details on the exploit, I'm pretty sure it's one of the following:

  • [Redacted for security]
  • [Redacted for security]

But by all means, I am no expert at this stuff so take the exploit guesses with a grain of salt. Thankfully, this isn't Go Daddy or 1 & 1. God knows what kind of PR garbage they would spit out and how long it would take them to actually fix the situation (I can assure you, it would probably a lot more than 2 days).

 

Also, if this ever happens again, could you tweet for clients to check the forums as I do not frequent them everyday? Thanks a lot Mike!

Link to comment
Share on other sites

*Sigh* A sad, sad day for MDD customers. It's good to know how hard Mike works to get these things fixed and is straight up honest with his clients. Also, if anyone want's specific details on the exploit, I'm pretty sure it's one of the following:

  • [Redacted for security]
  • [Redacted for security]

But by all means, I am no expert at this stuff so take the exploit guesses with a grain of salt. Thankfully, this isn't Go Daddy or 1 & 1. God knows what kind of PR garbage they would spit out and how long it would take them to actually fix the situation (I can assure you, it would probably a lot more than 2 days).

 

Official Operating System bug report and exploit details:

https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2010-3081

https://access.redhat.com/kb/docs/DOC-40265

 

We waited to post this information to make sure that our other services were secured against this exploit however these links will provide the technical details as to what has happened for any that may be interested. The last thing we wanted to do is post details on the exploit for another server to be compromised by the same exploit before it could be mitigated.

 

I did edit out your links as they linked directly to the code used to perpetrate the attack and as such we'd like to avoid disbursing exploitation code.

 

Also, if this ever happens again, could you tweet for clients to check the forums as I do not frequent them everyday? Thanks a lot Mike!

My best piece of advice is to subscribe to this forum section, we always make a thread if there is an issue. To be honest I should have tweeted but I was focused on identifying the attack, securing the other servers against the attack, and restoring the server that was compromised.
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...