Jump to content


Photo

[Resolved] Echo Server Repair


  • Please log in to reply
100 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 04:34 AM

Notice:
Normal service has been restored. You may skip ahead to the post announcing this: http://forums.mddhos...ndpost__p__1445

The remainder of this post is being left online for those who are interested. Once again, echo has been successfully restored and all services are back to normal.


------------------------

Hello,

Our Echo server was the target of zero-day kernel exploit. What this means in simple terms is that an attacker exploited a flaw in the operating system that is unknown to the operating system developers and as such there is not yet a patch for this issue.

Fortunately this is where our backup system comes into play as we're going to restore the server back 24 hours prior to the attack. The downside to this restoration is that it can easily take 10 to 20 hours for this process to complete. We've already started the restoration process so that we can get things back online as quickly as possible.

We have investigated the attack and determined that all the attacker has done is defaced index files (index.php, index.html, index.htm, etc...). We do not believe the attacker has stolen any data, dumped any databases, or modified any system files however we are going to do a full restore of the system just in case they left a back door into the system that we've not found.

We have taken some pro-active steps to help protect customers on other servers that are unaffected which includes disabling FTP and SSH for all accounts. For anybody on another server wishing to work on their sites we ask that you please use the "File Manager" from within your cPanel. We understand the frustration and trouble this may cause however it's better to have to take a couple of extra steps to work on your site than to face 10 to 20 hours of downtime or worse due to an unpatched exploit.

We always believe in being as open as possible about what is happening and what we're doing to resolve the issue. We do ask that if you have any questions about this issue to please post them in this thread so that we can centralize the disbursement of information to our client base.

We appreciate your understanding and patience in these trying times.

Update:
I'm posting this to the original post just to save new visitors to the thread from having to review the entire thread.

We're currently restoring the Operating System on the server back 24 hours prior to the exploit. Once the server OS is restored we're going to then begin restoring customer accounts from the backup system. This method will allow accounts to come online as they are restored instead of all accounts having to wait until the entire process is completed.

We're currently looking at approximately 48 hours for the last account to be restored however accounts will begin coming back online one at a time here within the next few hours.

We have no way of prioritizing which accounts are restored first or last and they will be restored as they are stored into the backup system. If you have any questions, do please read the rest of this thread and feel free to post your questions here.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#2 mipra

mipra

    Newbie

  • Members
  • Pip
  • 6 posts
  • Gender:Male
  • Location:Jakarta, Indonesia

Posted 17 September 2010 - 06:19 AM

That's honest enough for me.
There is nothing appriciated then pure honesty. I wish you all the best with the patch :blink:
  • 0

#3 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 06:23 AM

That's honest enough for me.
There is nothing appriciated then pure honesty. I wish you all the best with the patch :blink:

Hopefully it will be patched quickly, we'll update this thread and all outstanding support tickets if there are any changes. Right now we're waiting on the restoration on Echo to finish and for an official patch to be released.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#4 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 07:51 AM

A temporary way to mitigate the specific attack has been found and implemented while we wait on an official patch. We're re-enabling FTP and SSH on all servers however Echo is still restoring and will be for at least several more hours.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#5 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 08:20 AM

The restoration process is going a *lot* slower than we anticipated unfortunately. There is around 450 GB of data to restore and it looks as though it's going to take nearly 48 hours to restore unless the process speeds up somewhere along the way. Believe me - we don't want this to take any longer than it has to but I'd rather be upfront and honest about the time frame and it end up being much shorter than the other way around.

We do sincerely apologize for this downtime however we chose the route of keeping your data safe by restoring back to a time before the exploit to be sure that no malicious files/code was left on the server.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#6 Vitaliy

Vitaliy

    Newbie

  • Clients
  • Pip
  • 1 posts

Posted 17 September 2010 - 08:32 AM

You mentioned that the FTP is re-enabled on all servers. I just tried logging into my account, but without much luck. Is my site part of this hack or is it something else?
  • 0

#7 Mike_M

Mike_M

    Newbie

  • Members
  • Pip
  • 10 posts

Posted 17 September 2010 - 08:36 AM

Thanks for keeping us informed as always.

I wish I would of waited to upgrade my site until this weekend. Lot's of wasted hours for me. :blink:
  • 0

#8 mipra

mipra

    Newbie

  • Members
  • Pip
  • 6 posts
  • Gender:Male
  • Location:Jakarta, Indonesia

Posted 17 September 2010 - 09:19 AM

The restoration process is going a *lot* slower than we anticipated unfortunately. There is around 450 GB of data to restore and it looks as though it's going to take nearly 48 hours to restore unless the process speeds up somewhere along the way. Believe me - we don't want this to take any longer than it has to but I'd rather be upfront and honest about the time frame and it end up being much shorter than the other way around.

We do sincerely apologize for this downtime however we chose the route of keeping your data safe by restoring back to a time before the exploit to be sure that no malicious files/code was left on the server.

FTP tried and upped as of my post.
  • 0

#9 patlaw

patlaw

    Newbie

  • Clients
  • Pip
  • 9 posts
  • Gender:Male
  • Location:USA

Posted 17 September 2010 - 09:28 AM

Is email going to take 48 hours as well?
  • 1

#10 jrotunda85

jrotunda85

    Newbie

  • Members
  • Pip
  • 6 posts
  • Gender:Male
  • Location:Alexandria, VA

Posted 17 September 2010 - 10:06 AM

I certainly understand and appreciate the need for security; however, shutting down websites for 2 days is going to hurt a lot of people, especially people like myself who do a lot of business through the site. Is there anyway to temporarily restore these sites on functioning servers so there isn't 48 hours of downtime?
  • 0

#11 Mike_M

Mike_M

    Newbie

  • Members
  • Pip
  • 10 posts

Posted 17 September 2010 - 10:13 AM

I certainly understand and appreciate the need for security; however, shutting down websites for 2 days is going to hurt a lot of people, especially people like myself who do a lot of business through the site. Is there anyway to temporarily restore these sites on functioning servers so there isn't 48 hours of downtime?


I would like to now this as well. 48 hours with no access to my email and sites is not good at all.

MDD is top notch and I don't balme you for this at all but 48 hours without services is going to hurt alot of people.
  • 0

#12 Breakn3ck

Breakn3ck

    Newbie

  • Clients
  • Pip
  • 1 posts

Posted 17 September 2010 - 10:28 AM

Thanks for making this such a transparent process Mike. Best of luck with the patch and looking forward to getting my email services restored. :blink:
  • 0

#13 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 17 September 2010 - 10:51 AM

You mentioned that the FTP is re-enabled on all servers. I just tried logging into my account, but without much luck. Is my site part of this hack or is it something else?


If you are located on the echo server, then FTP would not be available as the server is still being restored. All other servers should have FTP back now. If your account is not on echo, please open a ticket at http://mddhosting.com/support so we may investigate for your account specifically.

Is email going to take 48 hours as well?


Unfortunately, yes. All services on echo, including email, are offline while the restore process is running.
  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#14 jonnyboy

jonnyboy

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 17 September 2010 - 11:29 AM

Will the email I've been sent while the server is down filter through once it comes back online, or will it be lost business?

I appreciate attacks like this happen, but hopefully the impact is minimal.
  • 0

#15 MjrNuT

MjrNuT

    Member

  • Clients
  • PipPip
  • 28 posts
  • Gender:Male

Posted 17 September 2010 - 11:36 AM

Thanks for the update MikeDVB.

Thanks for taking the precaution of removing access.

For clarification, you've noted a temporary mitigation has been determined. How is/was this tested?

Is the restore of Echo a bare metal level one?

Look forward to your reply when you've been able. I know you were up late....as I was. :blink:

Regards
  • 0
MjrNuT

#16 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 11:47 AM

I certainly understand and appreciate the need for security; however, shutting down websites for 2 days is going to hurt a lot of people, especially people like myself who do a lot of business through the site. Is there anyway to temporarily restore these sites on functioning servers so there isn't 48 hours of downtime?

What is taking so long is the actual restoration of the data - whether we restore it back to the same hardware (which is what we are doing) or we restored it back to a different server - it's going to take up to 48 hours to complete.

We don't want you to be offline any longer than you have to.


I would like to now this as well. 48 hours with no access to my email and sites is not good at all.

MDD is top notch and I don't balme you for this at all but 48 hours without services is going to hurt alot of people.

Indeed it is, on a positive note - if you do have your own cPanel backup of your account (I always personally recommend keeping your own off-provider backups, even with us) we can restore that account to an alternate server in the meantime. Open a support ticket and we'll get to work on these on a first come first served basis.

Will the email I've been sent while the server is down filter through once it comes back online, or will it be lost business?

I appreciate attacks like this happen, but hopefully the impact is minimal.

In most cases mail servers will continue trying to send emails for up to 72 hours, there are some cases where they will not and I cannot promise that everything will get through to you.


Thanks for the update MikeDVB.
Thanks for taking the precaution of removing access.
For clarification, you've noted a temporary mitigation has been determined. How is/was this tested?

The temporary mitigation was provided by the software vendor (RedHat) and was tested on a development box. The script used to exploit access to the server was tested before and after the mitigation was put in place on the development server and worked beautifully.

Is the restore of Echo a bare metal level one?

Yes.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#17 Mike_M

Mike_M

    Newbie

  • Members
  • Pip
  • 10 posts

Posted 17 September 2010 - 11:55 AM

I'm guessing that if I don't have my own Cpanel back up your SOL.
  • 0

#18 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 12:46 PM

I'm guessing that if I don't have my own Cpanel back up your SOL.

We have backups of your data, so you're not "SOL" in the sense that you could have been with another provider.

We're actually in contact with our backup vendor right now to see if we can get things restored any faster or if there is a better way to go about this. I'll see that this thread is updated if we change the action plan or anything else changes.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#19 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 12:58 PM

After speaking with our backup software provider they have advised us to take a different backup restoration route.

Here is what is happening:
  • We're restoring the operating system and system files
  • Bringing the server online
  • Restoring account files once the server is online.

The entire process is still going to take 48 hours however within about 5 hours from now sites will start to come back online. We have no way of choosing what accounts are restored first or last and it they will be restored in the order that they're stored in the backup system.

This will help prevent some additional downtime for a good portion of the clients affected and while we're still evaluating our options to get things back online more quickly this appears to be our best option at this point.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#20 Mike_M

Mike_M

    Newbie

  • Members
  • Pip
  • 10 posts

Posted 17 September 2010 - 01:15 PM

Thanks Mike!!! :blink:
  • 0




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users