Jump to content


Photo

[Resolved] Echo Server Repair


  • Please log in to reply
100 replies to this topic

#61 Mike_M

Mike_M

    Newbie

  • Members
  • Pip
  • 10 posts

Posted 18 September 2010 - 11:02 AM

We are all feeling the pain here in one form or another. This is just nerve wrecking.

Please provide us with a current update.

The natives are getting restless!!!
  • 0

#62 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,901 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 September 2010 - 11:38 AM

How much longer? Another day? Is it 10% done? 50%?

The timeframe hasn't change as of yet - we're still looking at approximately 2PM tomorrow (Sunday). As I said previously however this ETA is not set in stone as the backup system has been speeding up and slowing down due to fragmentation of the backup files on the backup server. Unfortunately the backup system is on an EXT3 File System right now which cannot be defragmented - we're moving to XFS which allows online defragmentation once this restoration is completed.

Was going on???? How much we have to wait we need a solution now is 24 hour off line or more, as many others we have clients under our services. How can I get my back ups to activate a provisional hosting and forward the domain with out chaging all my clients dns

You have clients under your services with us just as we have clients under our services with us - we're all in the same boat here. We're going to have it online as quickly as we can but unfortunately there's nothing more we can do to speed this up.

I still have no website and all of my emails have now disappeared from my mail client..please tell me that these will also be restored once the echo server comes back online?!

Your emails that were on the server up to the restoration point we're restoring to will come back, yes.

Michael and co., I'm understanding you to say that a load-balanced multi-server or full VPS configuration would not have prevented the present outage, correct?

Not necessarily - it really depends on how it was configured. Chances are in a load-balanced situation where an attacker gained root access to one server - the other one would quickly mirror those changes over and we'd be in the same position.

Can you clarify why only ECHO was affected? Is it simply because the attacker happened to have only targeted that machine, or are there differences (that you can at least reference here w/o putting those machines + clients at risk...)in the way in which those machines are configured?

The script that was used was uploaded to an account where a user failed to keep their script updated and secure. The user used a script exploit to place the file on the server and then used that same exploit to execute the script which did the damage.

The exploit that was used that did the damage itself was a zero-day exploit that was reported hours before it hit us and there was no known mitigation and no patch when we were hit by this. It's not something we could have prevented and it could have happened to any provider.

The thing you should all keep in mind is that while we do understand your frustration - things could be much worse. I don't know HOW many times I've seen, "Sorry, we lost your data due to XYZ and we don't have a backup of it. If you have your own backup file we'll be happy to restore your account." While the restoration process may be going much slower than we'd like - at least we have a backup copy of your data.

I've forwarded my domains while repairs are underway but just had a thought along the lines of the above. Could you have implemented the quick fix you mention above on a extra drive and then simply swapped drives once the backup transfer was complete? It would mean the risk of a backdoor but only so long as it would take to finish the backup. This would mean a hugely reduced downtime. It would also likely mean much more labour time for MDD I realize but I'm starting to notice how much money we're losing with regards to our ad campaigns and game sales as well as the fact that I'm starting to see our visitors posting on competing sites in search of a replacement for the resources we were providing.

It's not about labor - I can assure you we've been working more and harder over the last day than we have over the last year just doing our best to get things back online. Not only did we have to fully investigate the original issue and find a way to mitigate it but we also had to work on restoring the data.

We do have a backup of your data - it's just taking a long time to restore. Let me be a little clearer:
  • We've not lost your data, it's still safe.
  • It could be worse, your data could have been lost and many providers would have said, "Sorry, it's gone."
  • If you have your own off-provider backups - we'll be more than happy to restore them to another server.
  • No matter who you host with (be it us, or anybody else) you should always keep your own off provider backups.
  • R1Soft backups do not work in a way that we can simply "swap a drive" and have everybody back online.
  • We're absolutely not going to risk bringing a compromized server online, while the attacker didn't steal any data the first time around we're not going to risk having any of your data stolen this time around.
  • It's not smart to store personal information such as credit card numbers in a shared environment, but that doesn't stop people from doing it. Just one of the many reasons we're going with a full restore from before the exploit instead of allowing the possibility of re-exploitation.

Should I be feeling entitled to at least an MDD credit or is this simply the way of the net? I hate to put the pressure on but business is business after all. :(

Sure, you can have a one-month credit if that's what you desire - officially that's what our Terms of Service allow.


i can't believe The company don't have any contingency plan for this king of situations

We do have a contingency plan - we're restoring your data from a backup server. It could be worse - we could be like most other companies and not have a contingency plan and simply tell you, "Sorry, you're data is gone forever - do you have a backup?" We're not doing that, we're restoring your data - please be patient (I realize it's hard and frustrating, you have to imagine how we feel). The original situation was not something that could have been prevented as there are still no OS patches for this issue and the mitigation code (which does actually disable some OS functionality) was not released until after the Echo server was hit.


what happen with "A 99.9% uptime guarantee".

It's a monetary guarantee - you're welcome to request the one month credit that our terms of service state you would qualify for.

i can understand few hours off down time but 24 to 48 is a killer, i already lose some of my clients, the other ones are really upset

Read through this thread - I'm sure we're going to lose some clients to and I'm sure more than a few are really upset (including you). We're in the same position however we likely have more clients to lose and be upset due to this outage. Now don't get me wrong - I'm not trying to put you or your clients down... I'm just saying that we're in a bad situation as well.

i thanks that is weekend so the situation is not worse, waiting 24 to 48 is not a solution for all this, what is going to happens with all the data is been sent to the down server like mails, don't tell is loose i going to have a lot more problems.... i know is hard time for all of us, and i try to understand but how we going to be sure this is't happening again, the security is a main issue here, the company sell reliability and security and with this point we don't have any of dose. So i have to have 2 servers in order to provide my self with 99.99 uptime.

Or you could simply maintain your own backups of the accounts - we could have had you restored and back online in under an hour. It's a small price to pay if you can't stand to have any downtime at all. Again keep in mind that it could be worse.


Thanks for making this such a transparent process Mike.
You and your team must be exhausted and I am sure you are doing best you can handling the situation.

Thanks for keeping me/us in the loop..

BTW: I am sure some clients here get pissed off but their is no point to putt extra pressure on MDD. This can happen to any hosting.

Indeed it could have - I'm sure there are other providers that were hit by the same exploit - as I said it was an unpatched exploit that nobody know about until just before we were hit with it. You can think of it as standing in an open field on a clear and sunny day and then suddenly being struck my lightning. Nobody saw it coming and there was nothing that could have been done to prevent it. We're restoring the server back to before that happened but it's taking longer than we'd like.


We are all feeling the pain here in one form or another. This is just nerve wrecking.

Please provide us with a current update.

The natives are getting restless!!!

There's nothing to update - the process is still running and it's still on track for the ETA provided earlier.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#63 Mike_M

Mike_M

    Newbie

  • Members
  • Pip
  • 10 posts

Posted 18 September 2010 - 11:47 AM

Thanks for your quick responses Michael.

Could you tell us what the back up restore point will be? 24 hours prior, 48 hours...

Thanks again.
  • 0

#64 Chy

Chy

    Newbie

  • Members
  • Pip
  • 4 posts
  • Gender:Female
  • Location:El Paso, Texas
  • Interests:Addiction.

Posted 18 September 2010 - 11:57 AM

This may be an obvious answer but like most tired, frazzled and worried as well and need confirmation the restoration will take care of it. Members and clients reported viewing the hackers message prior to going offline so I know my files were infiltrated, will this be taken care of with the restoration? Or do I need to be concerned about going through and scrubbing files again on my domains? With the amount I have I dread the thought.
  • 0

#65 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,901 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 September 2010 - 12:02 PM

Thanks for your quick responses Michael.

Could you tell us what the back up restore point will be? 24 hours prior, 48 hours...

Thanks again.

09/13/2010 at 3:33:55 AM EST (GMT-5) is the point that is being used.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#66 Mike_M

Mike_M

    Newbie

  • Members
  • Pip
  • 10 posts

Posted 18 September 2010 - 12:12 PM

Why so far back? I'm going to be sick now. :(

Can we restore to another point once the server is back online?

Please tell me this can be done.
  • 0

#67 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,901 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 September 2010 - 12:16 PM

Why so far back? I'm going to be sick now. :(

Can we restore to another point once the server is back online?

Please tell me this can be done.

Some files were modified in the backup on the 14th from what we could determine - we're able to restore individual files/accounts/etc back to a newer date however we weren't going to restore the entire server to a newer date as not to risk a potential back-door.

If you have your own backup of your account we'll be happy to restore it to another server and get you online now.

Edited by MikeDVB, 18 September 2010 - 12:17 PM.
Corrected date, typo'd it.

  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#68 supernix

supernix

    Member

  • Clients
  • PipPip
  • 67 posts
  • Gender:Male
  • Location:South Carolina, USA

Posted 18 September 2010 - 12:17 PM

This sounds like our accounts were hosted on a Windows based computer. The whole feel secure nature of Linux is that it is a real multi-user environment. And with each users files set with the proper permissions it is completely impossible for one user to alter another's files.
But you just told us that some user with an account on the server didn't keep their scripts up to date and so now we all are paying a heavy price in a extremely long down time in computing terms.
For any exploit to work like you posted then that file would have to have been running with a privilege other than the user to which the file was executed.
This as best I can tell would mean that clients are not jail shelled and that the files are running under a common user account as say Apache.
Just trying to understand what happened and why.
  • 0
█ Cut Above Host
http://www.cutabovehost.com/
█ High Performance • Enterprise Servers • Premium Network
█ Great packages - Great Support - All around swell company
Future Crock Killer :-)

#69 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,901 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 September 2010 - 12:23 PM

This may be an obvious answer but like most tired, frazzled and worried as well and need confirmation the restoration will take care of it. Members and clients reported viewing the hackers message prior to going offline so I know my files were infiltrated, will this be taken care of with the restoration? Or do I need to be concerned about going through and scrubbing files again on my domains? With the amount I have I dread the thought.

We're restoring the server back to a time before any files were modified and the system was compromised.

This sounds like our accounts were hosted on a Windows based computer. The whole feel secure nature of Linux is that it is a real multi-user environment. And with each users files set with the proper permissions it is completely impossible for one user to alter another's files.
But you just told us that some user with an account on the server didn't keep their scripts up to date and so now we all are paying a heavy price in a extremely long down time in computing terms.

To be honest almost nobody keeps their scripts up to date and we see *accounts* compromised every day due to it and that's nothing new. What is new in this situation is a bug in the 2.6 Linux Kernel (the latest) that was exploited before there was a chance to develop a patch. I'm sorry if you've seen Linux as fool-proof and secure on all levels no matter what, as software is designed and written by humans and is therefore going to be flawed.

For any exploit to work like you posted then that file would have to have been running with a privilege other than the user to which the file was executed.

Which is exactly what the exploit did - it found a buffer underrun in the 64bit <-> 32bit conversion that allows 32 bit executables to run on a 64 bit server.

I've already linked to the details previously in this thread but in case you missed them, here they are again:
https://bugzilla.red...d=CVE-2010-3081
https://access.redha.../docs/DOC-40265

This as best I can tell would mean that clients are not jail shelled and that the files are running under a common user account as say Apache.
Just trying to understand what happened and why.

Very few clients have jailshell however they wouldn't have needed jailshell for this exploit to have worked as it did. Files run under the user name and not under "apache" or "nobody".

It's like I said - this was an exploit at the OS level (as per the bug report and the security report from RedHat - the developer of the system kernel) and ultimately there is *nothing* we could have done to prevent it.

A simple example is that if you buy a car brand new that's supposed to run for 100,000 miles and an axle breaks after 50 miles. It's a manufacturing fault in the AXLE and not *your* fault or the dealership's fault that it happened. The same idea applies here - this was a bug in the operating system core itself - there's nothing that us or our hardware provider could have done to prevent it.

Hopefully this clears some of your questions up - if not let me know.

Edit: If you are just now coming into this thread, please read it from the beginning and see if your questions have already been answered.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#70 iansltx

iansltx

    Member

  • Clients
  • PipPip
  • 32 posts

Posted 18 September 2010 - 12:28 PM

@supernix I can attest to the fact that MDDHosting is very definitely a Linux system...really hope you were kidding there.

Also, it's an exploit for crying out loud! When you get privilege escalation due to a flaw in the Linux kernel there isn't much you can't do. Again, the magnitude of this attack was the result of a kernel exploit, not a run-of-the-mill scripting vulnerability.

...and yes, my personal site and the site that I admin is down like everyone else's. Going to the office in a few minutes to see if I can grab a CPanel backup I made of the admin'd site so that can get online sooner rather than later.
  • 0

#71 Mike_M

Mike_M

    Newbie

  • Members
  • Pip
  • 10 posts

Posted 18 September 2010 - 12:33 PM

When exactly was they system compromised?
  • 0

#72 supernix

supernix

    Member

  • Clients
  • PipPip
  • 67 posts
  • Gender:Male
  • Location:South Carolina, USA

Posted 18 September 2010 - 12:43 PM

I was dumb enough when I started hosting in 2000 to use the IIS server and Windows 2000. Bad mistake as it was slapped around like a cheap ****** and owned by hackers when they were having the hacker wars back then. The only way to keep them out was to pull the plug.
And that is why I never host anything with Windows ever again.
  • 0
█ Cut Above Host
http://www.cutabovehost.com/
█ High Performance • Enterprise Servers • Premium Network
█ Great packages - Great Support - All around swell company
Future Crock Killer :-)

#73 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,901 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 September 2010 - 12:56 PM

I was dumb enough when I started hosting in 2000 to use the IIS server and Windows 2000. Bad mistake as it was slapped around like a cheap ****** and owned by hackers when they were having the hacker wars back then. The only way to keep them out was to pull the plug.
And that is why I never host anything with Windows ever again.

Linux can be the same way - it's all about whether you know what you're doing and how to properly secure your system. Even a secured system can have a system-level exploit (that you can't secure against) that can cause a major issue. It's fairly rare but they do happen.

If you just bring a server online with Linux, Apache, MySQL, and PHP without doing any work to secure it - it's going to end up compromised in short order, just like an unsecured Windows server.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#74 Dan S

Dan S

    Newbie

  • Clients
  • Pip
  • 4 posts

Posted 18 September 2010 - 01:00 PM

This can happen to any hosting.


Quick recovery from these issues is also possible for any hosting. Set expectations low and the results will follow.
  • 0

#75 MjrNuT

MjrNuT

    Member

  • Clients
  • PipPip
  • 28 posts
  • Gender:Male

Posted 18 September 2010 - 01:05 PM

I think you should roll some of your responses here into the OP for people when you have the time.

I have some follow up questions to some of your replies.



As stated previously, we don't need to see the details of the script used to perform this exploit.

Has the account owner been notified of their out of date "script" for which the exploit was injected? Can you divulge what the script and version is?

What has been done or going to be done with that account?

Was the account owner even aware of the injection or did they learn after access was removed to Echo altogether (i.e., all sites inaccessible)?


Are the other MDD servers configured the same as ECHO?



I am glad that MDD can commit to saying no data was lost.

I am glad that MDD has committed to a restore date of confidence (based on file changes), followed by potential of newer, specific restores on case-by-case basis.

I am glad that the communication is flowing given this technically, time-dependent resolution path.

Edited by MikeDVB, 18 September 2010 - 01:13 PM.
Removed extremely long quote to shorten the post and reduce duplicate content in the thread.

  • 0
MjrNuT

#76 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,901 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 September 2010 - 01:05 PM

Quick recovery from these issues is also possible for any hosting. Set expectations low and the results will follow.

Ever had something not go exactly how you planned? It's not that anybody is setting expectations low - it's that this is the real world. Our backup system is supposed to be able to bring a server back online from a restoration point in under 10 hours however it's performing slower than it should.

Our expectations of the backup system were very high and unfortunately in the backup system's current configuration it is excellent for small restorations such as individual accounts or databases but it's really suffering when it comes to restoring a whole server.

Due to us having high expectations - we've conferenced with our hardware vendors and R1Soft and come up with an action plan to put in place to ensure that if we ever have to restore a server again from a bare metal restoration point - that it *will* happen quickly.

As stated previously, we don't need to see the details of the script used to perform this exploit.

If you haven't been notified - then it wasn't you. Due to our respect for client privacy I'm not going to go into detail as to who it was.

Has the account owner been notified of their out of date "script" for which the exploit was injected? Can you divulge what the script and version is?

It's a WordPress script - it could be the script itself or one of the plugins (enabled or disabled) which was exploited. WordPress itself is updated regularly and I suggest keeping it up to date and only running well known and trusted plugins. I also recommend removing any unused plugins. While they're not called directly by WordPress - an exploit scanner can find them and execute them.

What has been done or going to be done with that account?

I don't see how this is a valid question - if there is an issue with your account I don't think you'd want me telling our entire client base what we're doing with your account.

Was the account owner even aware of the injection or did they learn after access was removed to Echo altogether (i.e., all sites inaccessible)?

Again, I don't see how this is a pertinent question. I understand your curiosity but I'm not going to discuss direct communications with a single client with the entire client base - at least - not without their permission.

Are the other MDD servers configured the same as ECHO?

Yes - the thing to keep in mind is that this exploit didn't rely upon poor configuration or security. It was a system core exploit - something that has to be patched against by the operating system developer (RedHat). Patches like these come out fairly regularly and we apply them as soon as they come out. We actually pay money for a system called KSplice to be able to install these system-level updates without having to reboot. If you are with another provider that isn't using a system similar to KSplice and isn't rebooting their servers monthly - chances are there are numerous system exploits just waiting to be taken advantage of.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#77 Nadir Novruzov

Nadir Novruzov

    Newbie

  • Clients
  • Pip
  • 15 posts

Posted 18 September 2010 - 01:19 PM

when sites will be online?
  • 0

#78 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,901 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 September 2010 - 01:28 PM

when sites will be online?

The answer:

The timeframe hasn't change as of yet - we're still looking at approximately 2PM tomorrow (Sunday). As I said previously however this ETA is not set in stone as the backup system has been speeding up and slowing down due to fragmentation of the backup files on the backup server. Unfortunately the backup system is on an EXT3 File System right now which cannot be defragmented - we're moving to XFS which allows online defragmentation once this restoration is completed.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#79 MjrNuT

MjrNuT

    Member

  • Clients
  • PipPip
  • 28 posts
  • Gender:Male

Posted 18 September 2010 - 01:34 PM

If you haven't been notified - then it wasn't you. Due to our respect for client privacy I'm not going to go into detail as to who it was.

It's a WordPress script - it could be the script itself or one of the plugins (enabled or disabled) which was exploited. WordPress itself is updated regularly and I suggest keeping it up to date and only running well known and trusted plugins. I also recommend removing any unused plugins. While they're not called directly by WordPress - an exploit scanner can find them and execute them.

I don't see how this is a valid question - if there is an issue with your account I don't think you'd want me telling our entire client base what we're doing with your account.

Again, I don't see how this is a pertinent question. I understand your curiosity but I'm not going to discuss direct communications with a single client with the entire client base - at least - not without their permission.

Yes - the thing to keep in mind is that this exploit didn't rely upon poor configuration or security. It was a system core exploit - something that has to be patched against by the operating system developer (RedHat). Patches like these come out fairly regularly and we apply them as soon as they come out. We actually pay money for a system called KSplice to be able to install these system-level updates without having to reboot. If you are with another provider that isn't using a system similar to KSplice and isn't rebooting their servers monthly - chances are there are numerous system exploits just waiting to be taken advantage of.

I was not intending to see details of the particular client be divulged. The questions were stated from the standpoint of reintroducing the problem again. I understand that a mitigation method will be put in place for the OS. Having said that in review of your reply, the compromised Wordpress script will resume, is that correct?

I was interested in where the source was as it is a part of the root-cause of the exploit. So I would think to disable that account such that the owner updates it, etc. before moving forward. This is the pertinence of the questions. I run Wordpress and I'd like to know if I was the idiot responsible and what needed to be done to fix it on behalf of others. :( I know I'm not the person since I was not notified.

I hope that clears up my questions and not intending for privacy to be ignored due to the event.
  • 0
MjrNuT

#80 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,901 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 September 2010 - 01:46 PM

I was not intending to see details of the particular client be divulged. The questions were stated from the standpoint of reintroducing the problem again. I understand that a mitigation method will be put in place for the OS. Having said that in review of your reply, the compromised Wordpress script will resume, is that correct?

The situation has been handled - it doesn't make sense for us to roll back the server just for us to allow it to become compromised again, do give us at least a little credit. :)

I was interested in where the source was as it is a part of the root-cause of the exploit. So I would think to disable that account such that the owner updates it, etc. before moving forward. This is the pertinence of the questions. I run Wordpress and I'd like to know if I was the idiot responsible and what needed to be done to fix it on behalf of others. :( I know I'm not the person since I was not notified.

The issue that allowed the code to be uploaded and run is separate from the code that actually caused the damage. Like I said - peoples sites get exploited every day on every provider due to not keeping scripts updated or not keeping a secure password and rotating it regularly. If the system core didn't have the bug that allowed the damage to happen - all that would have happened is the file would have been uploaded and it simply wouldn't have done anything (i.e. no damage).

I hope that clears up my questions and not intending for privacy to be ignored due to the event.

I understand what you were asking now and I've done my best to answer your question in detail without getting too technical.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users