
[Resolved] Echo Server Repair
#21
Posted 17 September 2010 - 01:42 PM
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#22
Posted 17 September 2010 - 02:42 PM
Thanks for the update, Mike!
#23
Posted 17 September 2010 - 02:44 PM
- [Redacted for security]
- [Redacted for security]
Also, if this ever happens again, could you tweet for clients to check the forums as I do not frequent them everyday? Thanks a lot Mike!
#24
Posted 17 September 2010 - 02:48 PM
*Sigh* A sad, sad day for MDD customers. It's good to know how hard Mike works to get these things fixed and is straight up honest with his clients. Also, if anyone want's specific details on the exploit, I'm pretty sure it's one of the following:
But by all means, I am no expert at this stuff so take the exploit guesses with a grain of salt. Thankfully, this isn't Go Daddy or 1 & 1. God knows what kind of PR garbage they would spit out and how long it would take them to actually fix the situation (I can assure you, it would probably a lot more than 2 days).
- [Redacted for security]
- [Redacted for security]
Official Operating System bug report and exploit details:
https://bugzilla.red...d=CVE-2010-3081
https://access.redha.../docs/DOC-40265
We waited to post this information to make sure that our other services were secured against this exploit however these links will provide the technical details as to what has happened for any that may be interested. The last thing we wanted to do is post details on the exploit for another server to be compromised by the same exploit before it could be mitigated.
I did edit out your links as they linked directly to the code used to perpetrate the attack and as such we'd like to avoid disbursing exploitation code.
My best piece of advice is to subscribe to this forum section, we always make a thread if there is an issue. To be honest I should have tweeted but I was focused on identifying the attack, securing the other servers against the attack, and restoring the server that was compromised.Also, if this ever happens again, could you tweet for clients to check the forums as I do not frequent them everyday? Thanks a lot Mike!
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#25
Posted 17 September 2010 - 04:02 PM
Assuming that update email actually finds its way into your Inbox. One would have to use an email address not attached to the server(s) having issues.My best piece of advice is to subscribe to this forum section, we always make a thread if there is an issue.
#26
Posted 17 September 2010 - 04:12 PM
Which is advised anyways - you generally do not want to use an email account hosted by a provider as a support contact for that provider no matter who the provider is.Assuming that update email actually finds its way into your Inbox. One would have to use an email address not attached to the server(s) having issues.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#27
Posted 17 September 2010 - 04:20 PM
#28
Posted 17 September 2010 - 04:24 PM
Which is advised anyways - you generally do not want to use an email account hosted by a provider as a support contact for that provider no matter who the provider is.
With Gmail accounts being free and easily accessible by POP and IMAP, it makes sense to have a spare email account for such purposes. Create an account, configure it for POP access on your main email client, and use it as the contact email for your server contact. That's what I do with pobox.com, which is an email aliasing services. When my echo POP box died, I pointed pobox and my email client to a different POP box, so at least that account is operating normally.
#29
Posted 17 September 2010 - 04:28 PM
Often times what is more important than what happens, is how it is handled. It's best to let our clients know exactly what is happening, why it happened, and what is being done about it. We've planned some major changes for our backup system moving forward from this issue so should we ever have to do anything like this again the restore should be magnitudes faster.Just wanted to comment in support of the open and responsive way this is being handled. This is the ONLY time I've observed unscheduled downtime in over a year of using the service, and in contrast to the extremely poor way a similar situation was handled by another one of our hosting providers a few weeks ago - this is a breath of fresh air!
The restoration process is still in progress, I'll update the thread if anything changes.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#30
Posted 17 September 2010 - 04:36 PM
#31
Posted 17 September 2010 - 04:39 PM
We're looking at somewhere around 10 PM ~ Midnight EST (GMT-5) for the server to be online where we can begin restoring account data. Every time I've posted an ETA it is getting extended due to the backup system continually slowing down more and more.Please start including an ETA, if possible. One of my hosting clients is an attorney who is totally non-technical. It takes an hour to walk her through setup changes in her email client. I'd rather not have to go through the process of moving her to a new server. She has to use the echo IP address because of problems she was having with her ISP.
None of the times posted in this thread are set in stone and are best educated guesses based upon the rate of transfer and the data left to transfer. Things could pick up significantly 1/2 way through the restore and finish way ahead of schedule or slow down even more and take substantially longer.
Right now what is being restored is the OS and MySQL databases as the MySQL databases are stored on a partition with system information required for booting. Once the server is online we'll begin restoring the partition which contains all of the hosting files (e.g. your /public_html and email accounts).
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#32
Posted 17 September 2010 - 05:37 PM
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#33
Posted 17 September 2010 - 06:45 PM
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#34
Posted 17 September 2010 - 07:11 PM

Is there something you can do to inform our visitors of the situation? A basic notice page for all domain names associated with the Echo server would be a nice temporary means. Maybe a temp server could do this while Echo is being restored?
Otherwise, should we temp forward our domains or would it be pointless considering propagation?
Thank you!
#35
Posted 17 September 2010 - 07:16 PM
I've considered setting something like this up however to do so we'd have to edit every DNS zone on the server to point to another server and then when it came time to power Echo back on all of this work would have to manually be reversed which would result in even more downtime even once the server is back online.Heya Michael. Great job you are doing and thank you for providing us with a great deal of feedback on the situation. At this point I'm wondering what to do in regards to informing my members and readers of what is going on. At the moment there is nothing that informs the visitors of the Echo websites about what is going on. For all our visitors know we all just packed up and left which leaves me worrying about my visitation numbers. 3 days down is a painfully long time.
![]()
Is there something you can do to inform our visitors of the situation? A basic notice page for all domain names associated with the Echo server would be a nice temporary means. Maybe a temp server could do this while Echo is being restored?
Otherwise, should we temp forward our domains or would it be pointless considering propagation?
Thank you!
If you do have a cPanel backup of your account we can restore it to another one of our servers and bring you online at least partially between now and when the restoration is completed.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#36
Posted 17 September 2010 - 07:22 PM
I have had my own domain since May, 1998 and I have never experienced a hosting outage of this magnitude. I work in IT and my boss' head would be on the platter if a server was down this long, whether it served internal or external customers. Where's the redundancy?Unfortunately doing the restore with the server online isn't going to be feasible, we're having to do a full restore of the server from start to finish before brining any accounts online. This process is going to take the full 24 to 48 hours and we'll update you if there are any changes.
The open, honest communication is great, but...
#37
Posted 17 September 2010 - 07:22 PM
#38
Posted 17 September 2010 - 07:23 PM
Just curious, whats the default TTL of domains on MDDHosting's DNS? Maybe they should be reduced for the time being in case anyone wants to point to a different server temporarily.
#39
Posted 17 September 2010 - 07:25 PM
Hmmm, ok. I think I have a full backup somewhere that we can make due with in the meantime. I know I have 5 day old database backups for sure. Should I ticket MDD?
If you would like to use this backup to be restored to a new server, open a ticket and we can make that happen for you. If you would rather wait for the more up-to-date backup, then I would suggest waiting.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#40
Posted 17 September 2010 - 07:26 PM
If you want a load-balanced fail-over setup that wouldn't be affected by this type of issue you'd be paying $75+/month just for a shared account. The issue there is that it really depends on how the data is replicated across the balancer as to whether that would even help in an exploitation situation. It'd save your day in the event of hardware failure but in a system-level root kernel exploit it would be helpless to prevent the damage.I have had my own domain since May, 1998 and I have never experienced a hosting outage of this magnitude. I work in IT and my boss' head would be on the platter if a server was down this long, whether it served internal or external customers. Where's the redundancy?
This was a zero-day kernel exploit - I'll be honest that we *probably* could have just restored the defaced/deleted data and gone from there and had much less downtime but my question is this: What happens if we take this shortcut and then the server is re-exploited through a hidden back-door and your data is not only lost but stolen?
It's not a risk we're willing to take - we're restoring the server back to a point in time before the attack happened to be sure that the server is secure and we're going to mitigate the exploit before bringing the public network online.
If you have your own backup of your account, open a ticket and we'll restore it to another server and get you back online very quickly. If you don't, you're going to have to wait for the server to be restored from the backup.
Our backup system is supposed to be able to do a full server restore in 5 to 10 hours however due to unforeseen circumstances it's taking substantially longer. We've done everything we can to speed up the process however there is only so much that can be done at this point.
If you do have any further questions you're welcome to post them.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users