Jump to content


Photo

[Resolved] Echo Server Repair


  • Please log in to reply
100 replies to this topic

#41 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 07:29 PM

Just curious, whats the default TTL of domains on MDDHosting's DNS? Maybe they should be reduced for the time being in case anyone wants to point to a different server temporarily.

14400 is the default TTL.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#42 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 17 September 2010 - 07:36 PM

Heya Michael. Great job you are doing and thank you for providing us with a great deal of feedback on the situation. At this point I'm wondering what to do in regards to informing my members and readers of what is going on. At the moment there is nothing that informs the visitors of the Echo websites about what is going on. For all our visitors know we all just packed up and left which leaves me worrying about my visitation numbers. 3 days down is a painfully long time. :(

Is there something you can do to inform our visitors of the situation? A basic notice page for all domain names associated with the Echo server would be a nice temporary means. Maybe a temp server could do this while Echo is being restored?

Otherwise, should we temp forward our domains or would it be pointless considering propagation?

Thank you!

I've considered setting something like this up however to do so we'd have to edit every DNS zone on the server to point to another server and then when it came time to power Echo back on all of this work would have to manually be reversed which would result in even more downtime even once the server is back online.

If you do have a cPanel backup of your account we can restore it to another one of our servers and bring you online at least partially between now and when the restoration is completed.

Hmmm, ok. I think I have a full backup somewhere that we can make due with in the meantime. I know I have 5 day old database backups for sure. Should I ticket MDD?

If you would like to use this backup to be restored to a new server, open a ticket and we can make that happen for you. If you would rather wait for the more up-to-date backup, then I would suggest waiting.


Hmmm, Well I'd like both actually. I was thinking that we could put my old backup, from February, up so that I could at least change my indexes to "Server Down" notices in order to inform my visitors. Then once you have completed restoration of Echo you could switch my account from my old Backup to your backup since it would be far more current.
  • 0

#43 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 07:50 PM

Hmmm, Well I'd like both actually. I was thinking that we could put my old backup, from February, up so that I could at least change my indexes to "Server Down" notices in order to inform my visitors. Then once you have completed restoration of Echo you could switch my account from my old Backup to your backup since it would be far more current.

Absolutely - anybody who has a backup of their account we can bring online on our Fresco server, we're noting which accounts we're doing this for and are going to reverse the changes once Echo is back online.

Do keep in mind that this may cause additional downtime for your up-to-date site from Echo due to DNS Propagation switching back from Fresco to Echo.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#44 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 17 September 2010 - 08:00 PM

Ah thanx Michael. That's exactly what I had in mind. Just one last question. My domains are with GoDaddy. I have a portal page hosted with a different company and is relative to the sites that are on Echo.

I could Temp forward the domains to that site hosted elsewhere and post the "Server Down" notice there.
or
Do as mentioned above with regards to using Fresco and then switching back to Echo.

Which option do you think would be better? I'm really not sure...
  • 0

#45 supernix

supernix

    Member

  • Clients
  • PipPip
  • 67 posts
  • Gender:Male
  • Location:South Carolina, USA

Posted 17 September 2010 - 08:01 PM

I would think that the backups would have been in the same data center.
That way you just pull the old drive and slap in the new.
  • 0
█ Cut Above Host
http://www.cutabovehost.com/
█ High Performance • Enterprise Servers • Premium Network
█ Great packages - Great Support - All around swell company
Future Crock Killer :-)

#46 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 08:09 PM

Ah thanx Michael. That's exactly what I had in mind. Just one last question. My domains are with GoDaddy. I have a portal page hosted with a different company and is relative to the sites that are on Echo.

I could Temp forward the domains to that site hosted elsewhere and post the "Server Down" notice there.
or
Do as mentioned above with regards to using Fresco and then switching back to Echo.

Which option do you think would be better? I'm really not sure...

Using an existing account instead of having a new one created would be ideal as we won't have to fix your DNS records once Echo is back online on the servers themselves.


I would think that the backups would have been in the same data center.
That way you just pull the old drive and slap in the new.

R1Soft backups unfortunately don't work like that - if they did we'd have no way to store multiple restoration points. We can restore files, databases, and whole servers as far back as we choose to keep restoration points with R1Soft where as simply mirroring the content daily to another server wouldn't allow this although restoration would be quicker if we needed to do it, assuming we got to it before the next copy ran and copied over the bad versions.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#47 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 17 September 2010 - 08:28 PM

Ok perfect. Thanx Michael. I'm going to temp forward to my other site until this is over.

Good luck on the clean up and may your pot of coffee be of plenty.
  • 1

#48 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 08:33 PM

Ok perfect. Thanx Michael. I'm going to temp forward to my other site until this is over.

Good luck on the clean up and may your pot of coffee be of plenty.

Scott and Dan have been covering this issue for me so I can do things like taking a nap or going to the bathroom. Since the event occurred I've only managed to get about 2 hours of sleep. It's too bad I'm not a huge fan of coffee.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#49 iansltx

iansltx

    Member

  • Clients
  • PipPip
  • 32 posts

Posted 17 September 2010 - 11:07 PM

I have a recent CPanel backup for one of the sites that I admin that *was* on Echo. I'll probably be able to get to it tomorrow. For now, I'd like to get a "the system is down" page up on Fresco. Should I just submit a ticket to get this done?
  • 0

#50 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 September 2010 - 11:17 PM

I have a recent CPanel backup for one of the sites that I admin that *was* on Echo. I'll probably be able to get to it tomorrow. For now, I'd like to get a "the system is down" page up on Fresco. Should I just submit a ticket to get this done?

Yes, a ticket would be the way to go.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#51 weetu

weetu

    Newbie

  • Clients
  • Pip
  • 18 posts

Posted 18 September 2010 - 04:50 AM

How much longer? Another day? Is it 10% done? 50%?
  • 0

#52 Betacentauro

Betacentauro

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 18 September 2010 - 06:43 AM

Was going on???? How much we have to wait we need a solution now is 24 hour off line or more, as many others we have clients under our services. How can I get my back ups to activate a provisional hosting and forward the domain with out chaging all my clients dns

Thanks
  • 0

#53 patlaw

patlaw

    Newbie

  • Clients
  • Pip
  • 9 posts
  • Gender:Male
  • Location:USA

Posted 18 September 2010 - 08:33 AM

What's going on?

Read this thread starting at the beginning. Everything is explained in detail.

How much we have to wait we need a solution now is 24 hour off line or more, as many others we have clients under our services?

Everything is explained above.

How can I get my back ups to activate a provisional hosting and forward the domain with out chaging all my clients dns?

As mentioned in the thread above, if you have your backups, MDD will install them on another server and configure the DNS. If you don't have backups, you will have to wait for the restore at MDD to finish.
  • 0

#54 Mike_M

Mike_M

    Newbie

  • Members
  • Pip
  • 10 posts

Posted 18 September 2010 - 08:54 AM

How much longer? Another day? Is it 10% done? 50%?



+1 :(
  • 0

#55 jonnyboy

jonnyboy

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 18 September 2010 - 08:59 AM

I still have no website and all of my emails have now disappeared from my mail client..please tell me that these will also be restored once the echo server comes back online?!
  • 0

#56 patlaw

patlaw

    Newbie

  • Clients
  • Pip
  • 9 posts
  • Gender:Male
  • Location:USA

Posted 18 September 2010 - 09:13 AM

As of 6:45PM yesterday...

This process is going to take the full 24 to 48 hours and we'll update you if there are any changes.


Therefore, the prediction as of yesterday is that we will be back in service tonight or tomorrow night.
  • 0

#57 Michael J. Russell

Michael J. Russell

    Newbie

  • Clients
  • Pip
  • 1 posts

Posted 18 September 2010 - 10:06 AM

Michael and co., I'm understanding you to say that a load-balanced multi-server or full VPS configuration would not have prevented the present outage, correct?

Can you clarify why only ECHO was affected? Is it simply because the attacker happened to have only targeted that machine, or are there differences (that you can at least reference here w/o putting those machines + clients at risk...)in the way in which those machines are configured?

Please advise. TIA.
  • 0

#58 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 18 September 2010 - 10:23 AM

If you want a load-balanced fail-over setup that wouldn't be affected by this type of issue you'd be paying $75+/month just for a shared account. The issue there is that it really depends on how the data is replicated across the balancer as to whether that would even help in an exploitation situation. It'd save your day in the event of hardware failure but in a system-level root kernel exploit it would be helpless to prevent the damage.

This was a zero-day kernel exploit - I'll be honest that we *probably* could have just restored the defaced/deleted data and gone from there and had much less downtime but my question is this: What happens if we take this shortcut and then the server is re-exploited through a hidden back-door and your data is not only lost but stolen?

It's not a risk we're willing to take - we're restoring the server back to a point in time before the attack happened to be sure that the server is secure and we're going to mitigate the exploit before bringing the public network online.

If you have your own backup of your account, open a ticket and we'll restore it to another server and get you back online very quickly. If you don't, you're going to have to wait for the server to be restored from the backup.

Our backup system is supposed to be able to do a full server restore in 5 to 10 hours however due to unforeseen circumstances it's taking substantially longer. We've done everything we can to speed up the process however there is only so much that can be done at this point.

If you do have any further questions you're welcome to post them.


I've forwarded my domains while repairs are underway but just had a thought along the lines of the above. Could you have implemented the quick fix you mention above on a extra drive and then simply swapped drives once the backup transfer was complete? It would mean the risk of a backdoor but only so long as it would take to finish the backup. This would mean a hugely reduced downtime. It would also likely mean much more labour time for MDD I realize but I'm starting to notice how much money we're losing with regards to our ad campaigns and game sales as well as the fact that I'm starting to see our visitors posting on competing sites in search of a replacement for the resources we were providing.

Should I be feeling entitled to at least an MDD credit or is this simply the way of the net? I hate to put the pressure on but business is business after all. :(
  • 0

#59 Betacentauro

Betacentauro

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 18 September 2010 - 10:36 AM

i can't believe The company don't have any contingency plan for this king of situations, what happen with "A 99.9% uptime guarantee". i can understand few hours off down time but 24 to 48 is a killer, i already lose some of my clients, the other ones are really upset, i thanks that is weekend so the situation is not worse, waiting 24 to 48 is not a solution for all this, what is going to happens with all the data is been sent to the down server like mails, don't tell is loose i going to have a lot more problems.... i know is hard time for all of us, and i try to understand but how we going to be sure this is't happening again, the security is a main issue here, the company sell reliability and security and with this point we don't have any of dose. So i have to have 2 servers in order to provide my self with 99.99 uptime.
  • 0

#60 Ivone

Ivone

    Newbie

  • Clients
  • Pip
  • 12 posts
  • Gender:Male
  • Location:Sunny Costa Rica
  • Interests:aXn

Posted 18 September 2010 - 11:00 AM

Thanks for making this such a transparent process Mike.
You and your team must be exhausted and I am sure you are doing best you can handling the situation.

Thanks for keeping me/us in the loop..

BTW: I am sure some clients here get pissed off but their is no point to putt extra pressure on MDD. This can happen to any hosting.
  • 0




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users