[Resolved] Echo Server Repair

[Michael D.]

Assuming that update email actually finds its way into your Inbox. One would have to use an email address not attached to the server(s) having issues.

Which is advised anyways - you generally do not want to use an email account hosted by a provider as a support contact for that provider no matter who the provider is.

[karrinina]

Just wanted to comment in support of the open and responsive way this is being handled. This is the ONLY time I've observed unscheduled downtime in over a year of using the service, and in contrast to the extremely poor way a similar situation was handled by another one of our hosting providers a few weeks ago - this is a breath of fresh air!

[patlaw]

Which is advised anyways - you generally do not want to use an email account hosted by a provider as a support contact for that provider no matter who the provider is.

 

With Gmail accounts being free and easily accessible by POP and IMAP, it makes sense to have a spare email account for such purposes. Create an account, configure it for POP access on your main email client, and use it as the contact email for your server contact. That's what I do with pobox.com, which is an email aliasing services. When my echo POP box died, I pointed pobox and my email client to a different POP box, so at least that account is operating normally.

[Michael D.]

Just wanted to comment in support of the open and responsive way this is being handled. This is the ONLY time I've observed unscheduled downtime in over a year of using the service, and in contrast to the extremely poor way a similar situation was handled by another one of our hosting providers a few weeks ago - this is a breath of fresh air!

Often times what is more important than what happens, is how it is handled. It's best to let our clients know exactly what is happening, why it happened, and what is being done about it. We've planned some major changes for our backup system moving forward from this issue so should we ever have to do anything like this again the restore should be magnitudes faster.

 

The restoration process is still in progress, I'll update the thread if anything changes.

[patlaw]

Please start including an ETA, if possible. One of my hosting clients is an attorney who is totally non-technical. It takes an hour to walk her through setup changes in her email client. I'd rather not have to go through the process of moving her to a new server. She has to use the echo IP address because of problems she was having with her ISP.

[Michael D.]

Please start including an ETA, if possible. One of my hosting clients is an attorney who is totally non-technical. It takes an hour to walk her through setup changes in her email client. I'd rather not have to go through the process of moving her to a new server. She has to use the echo IP address because of problems she was having with her ISP.

We're looking at somewhere around 10 PM ~ Midnight EST (GMT-5) for the server to be online where we can begin restoring account data. Every time I've posted an ETA it is getting extended due to the backup system continually slowing down more and more.

 

None of the times posted in this thread are set in stone and are best educated guesses based upon the rate of transfer and the data left to transfer. Things could pick up significantly 1/2 way through the restore and finish way ahead of schedule or slow down even more and take substantially longer.

 

Right now what is being restored is the OS and MySQL databases as the MySQL databases are stored on a partition with system information required for booting. Once the server is online we'll begin restoring the partition which contains all of the hosting files (e.g. your /public_html and email accounts).

[Michael D.]

The backup speed picked up quite a bit - if the speed continues the server itself will be online within an hour and we'll begin brining accounts online.

[Michael D.]

Unfortunately doing the restore with the server online isn't going to be feasible, we're having to do a full restore of the server from start to finish before brining any accounts online. This process is going to take the full 24 to 48 hours and we'll update you if there are any changes.

[Brad]

Heya Michael. Great job you are doing and thank you for providing us with a great deal of feedback on the situation. At this point I'm wondering what to do in regards to informing my members and readers of what is going on. At the moment there is nothing that informs the visitors of the Echo websites about what is going on. For all our visitors know we all just packed up and left which leaves me worrying about my visitation numbers. 3 days down is a painfully long time.

 

Is there something you can do to inform our visitors of the situation? A basic notice page for all domain names associated with the Echo server would be a nice temporary means. Maybe a temp server could do this while Echo is being restored?

 

Otherwise, should we temp forward our domains or would it be pointless considering propagation?

 

Thank you!

[Michael D.]

Heya Michael. Great job you are doing and thank you for providing us with a great deal of feedback on the situation. At this point I'm wondering what to do in regards to informing my members and readers of what is going on. At the moment there is nothing that informs the visitors of the Echo websites about what is going on. For all our visitors know we all just packed up and left which leaves me worrying about my visitation numbers. 3 days down is a painfully long time.

 

Is there something you can do to inform our visitors of the situation? A basic notice page for all domain names associated with the Echo server would be a nice temporary means. Maybe a temp server could do this while Echo is being restored?

 

Otherwise, should we temp forward our domains or would it be pointless considering propagation?

 

Thank you!

I've considered setting something like this up however to do so we'd have to edit every DNS zone on the server to point to another server and then when it came time to power Echo back on all of this work would have to manually be reversed which would result in even more downtime even once the server is back online.

 

If you do have a cPanel backup of your account we can restore it to another one of our servers and bring you online at least partially between now and when the restoration is completed.

[Dan S]

Unfortunately doing the restore with the server online isn't going to be feasible, we're having to do a full restore of the server from start to finish before brining any accounts online. This process is going to take the full 24 to 48 hours and we'll update you if there are any changes.

I have had my own domain since May, 1998 and I have never experienced a hosting outage of this magnitude. I work in IT and my boss' head would be on the platter if a server was down this long, whether it served internal or external customers. Where's the redundancy?

 

The open, honest communication is great, but...

[Brad]

Hmmm, ok. I think I have a full backup somewhere that we can make due with in the meantime. I know I have 5 day old database backups for sure. Should I ticket MDD?

[kocchi]

Otherwise, your next best bet is to sign up for a temporary host somewhere and change your DNS Host to them (or point to their servers, depending on if you could manage the DNS), and put up a message there. Make sure to keep your Time-to-Live as low as possible.

 

Just curious, whats the default TTL of domains on MDDHosting's DNS? Maybe they should be reduced for the time being in case anyone wants to point to a different server temporarily.

[Scott]

Hmmm, ok. I think I have a full backup somewhere that we can make due with in the meantime. I know I have 5 day old database backups for sure. Should I ticket MDD?

 

If you would like to use this backup to be restored to a new server, open a ticket and we can make that happen for you. If you would rather wait for the more up-to-date backup, then I would suggest waiting.

[Michael D.]

I have had my own domain since May, 1998 and I have never experienced a hosting outage of this magnitude. I work in IT and my boss' head would be on the platter if a server was down this long, whether it served internal or external customers. Where's the redundancy?

If you want a load-balanced fail-over setup that wouldn't be affected by this type of issue you'd be paying $75+/month just for a shared account. The issue there is that it really depends on how the data is replicated across the balancer as to whether that would even help in an exploitation situation. It'd save your day in the event of hardware failure but in a system-level root kernel exploit it would be helpless to prevent the damage.

 

This was a zero-day kernel exploit - I'll be honest that we *probably* could have just restored the defaced/deleted data and gone from there and had much less downtime but my question is this: What happens if we take this shortcut and then the server is re-exploited through a hidden back-door and your data is not only lost but stolen?

 

It's not a risk we're willing to take - we're restoring the server back to a point in time before the attack happened to be sure that the server is secure and we're going to mitigate the exploit before bringing the public network online.

 

If you have your own backup of your account, open a ticket and we'll restore it to another server and get you back online very quickly. If you don't, you're going to have to wait for the server to be restored from the backup.

 

Our backup system is supposed to be able to do a full server restore in 5 to 10 hours however due to unforeseen circumstances it's taking substantially longer. We've done everything we can to speed up the process however there is only so much that can be done at this point.

 

If you do have any further questions you're welcome to post them.

[Michael D.]

Just curious, whats the default TTL of domains on MDDHosting's DNS? Maybe they should be reduced for the time being in case anyone wants to point to a different server temporarily.

14400 is the default TTL.

[Brad]

Heya Michael. Great job you are doing and thank you for providing us with a great deal of feedback on the situation. At this point I'm wondering what to do in regards to informing my members and readers of what is going on. At the moment there is nothing that informs the visitors of the Echo websites about what is going on. For all our visitors know we all just packed up and left which leaves me worrying about my visitation numbers. 3 days down is a painfully long time.

 

Is there something you can do to inform our visitors of the situation? A basic notice page for all domain names associated with the Echo server would be a nice temporary means. Maybe a temp server could do this while Echo is being restored?

 

Otherwise, should we temp forward our domains or would it be pointless considering propagation?

 

Thank you!

I've considered setting something like this up however to do so we'd have to edit every DNS zone on the server to point to another server and then when it came time to power Echo back on all of this work would have to manually be reversed which would result in even more downtime even once the server is back online.

 

If you do have a cPanel backup of your account we can restore it to another one of our servers and bring you online at least partially between now and when the restoration is completed.

Hmmm, ok. I think I have a full backup somewhere that we can make due with in the meantime. I know I have 5 day old database backups for sure. Should I ticket MDD?

If you would like to use this backup to be restored to a new server, open a ticket and we can make that happen for you. If you would rather wait for the more up-to-date backup, then I would suggest waiting.

 

Hmmm, Well I'd like both actually. I was thinking that we could put my old backup, from February, up so that I could at least change my indexes to "Server Down" notices in order to inform my visitors. Then once you have completed restoration of Echo you could switch my account from my old Backup to your backup since it would be far more current.

[Michael D.]

Hmmm, Well I'd like both actually. I was thinking that we could put my old backup, from February, up so that I could at least change my indexes to "Server Down" notices in order to inform my visitors. Then once you have completed restoration of Echo you could switch my account from my old Backup to your backup since it would be far more current.

Absolutely - anybody who has a backup of their account we can bring online on our Fresco server, we're noting which accounts we're doing this for and are going to reverse the changes once Echo is back online.

 

Do keep in mind that this may cause additional downtime for your up-to-date site from Echo due to DNS Propagation switching back from Fresco to Echo.

[Brad]

Ah thanx Michael. That's exactly what I had in mind. Just one last question. My domains are with GoDaddy. I have a portal page hosted with a different company and is relative to the sites that are on Echo.

 

I could Temp forward the domains to that site hosted elsewhere and post the "Server Down" notice there.

or

Do as mentioned above with regards to using Fresco and then switching back to Echo.

 

Which option do you think would be better? I'm really not sure...

[supernix]

I would think that the backups would have been in the same data center.

That way you just pull the old drive and slap in the new.

[Michael D.]

Ah thanx Michael. That's exactly what I had in mind. Just one last question. My domains are with GoDaddy. I have a portal page hosted with a different company and is relative to the sites that are on Echo.

 

I could Temp forward the domains to that site hosted elsewhere and post the "Server Down" notice there.

or

Do as mentioned above with regards to using Fresco and then switching back to Echo.

 

Which option do you think would be better? I'm really not sure...

Using an existing account instead of having a new one created would be ideal as we won't have to fix your DNS records once Echo is back online on the servers themselves.

 

 

I would think that the backups would have been in the same data center.

That way you just pull the old drive and slap in the new.

R1Soft backups unfortunately don't work like that - if they did we'd have no way to store multiple restoration points. We can restore files, databases, and whole servers as far back as we choose to keep restoration points with R1Soft where as simply mirroring the content daily to another server wouldn't allow this although restoration would be quicker if we needed to do it, assuming we got to it before the next copy ran and copied over the bad versions.

[Brad]

Ok perfect. Thanx Michael. I'm going to temp forward to my other site until this is over.

 

Good luck on the clean up and may your pot of coffee be of plenty.

[Michael D.]

Ok perfect. Thanx Michael. I'm going to temp forward to my other site until this is over.

 

Good luck on the clean up and may your pot of coffee be of plenty.

Scott and Dan have been covering this issue for me so I can do things like taking a nap or going to the bathroom. Since the event occurred I've only managed to get about 2 hours of sleep. It's too bad I'm not a huge fan of coffee.

[iansltx]

I have a recent CPanel backup for one of the sites that I admin that *was* on Echo. I'll probably be able to get to it tomorrow. For now, I'd like to get a "the system is down" page up on Fresco. Should I just submit a ticket to get this done?

[Michael D.]

I have a recent CPanel backup for one of the sites that I admin that *was* on Echo. I'll probably be able to get to it tomorrow. For now, I'd like to get a "the system is down" page up on Fresco. Should I just submit a ticket to get this done?

Yes, a ticket would be the way to go.