Resolved DDoS Attack on Echo / 173.248.188.110

[Michael D.]

Hello,

 

Unfortunately the Echo Server came under a DDoS attack this morning. We've been working to mitigate this attack, however, it seems that no matter what we've tried this attack is simply too large to be mitigated.

 

I apologize for the lack in details over the last hour - our night shift was working to handle the issue and they chose not to alert management to the issue as they were attempting to resolve the issue without 'bothering' management. I've made it clear that for service affecting incidents that management be alerted without exception in the future.

 

We're now working to split accounts off of the affected IP to restore service to as many customers as we can as quickly as possible but this process does take some time.

[kix766]

hi mike

 

since this attack can we say that emails inbound will not be effected ? since the attack no emails have been received..

[Michael D.]

IP changes are still in progress. If you are affected you can log into your cPanel at http://echo.supportedns.com/cpanel and you can check/send email by making sure you are connecting to "echo.supportedns.com" and not "mail.yourdomain.com".

 

If you are not using our DNS [generally you would know if this is the case - if you do not know, then you are most likely using our DNS] you will need to update your third party DNS once your IP changes. On the left side under 'statistics' you can see your 'Shared IP Address' [you may need to hit 'expand']. Once you see it change from 173.248.188.110 to something different you would want to update your third party DNS.

 

Understand that due to the nature of this attack and the number of accounts affected I cannot guarantee that the IP you are moved to is the one that you will remain on - this is one of the reasons we advise using our DNS as any DNS updates with IP changes while using our DNS are automatic.

[Michael D.]

hi mike

 

since this attack can we say that emails inbound will not be effected ? since the attack no emails have been received..

The email server is still working just fine - just update your mail client to connect to "echo.supportedns.com" - you can also check it via webmail at http://echo.supportedns.com/webmail.

 

Inbound email may be delayed, however, due to the nature of mail servers any missing mail should be delivered within about an hour of service being fully restored to the affected accounts.

[kix766]

thanks. everything is set to echo.supportedns.com already but no emails have been received since we repoted the attack.

 

Hopefully when restored they will all come in at once.

 

Should we trace back with clients in case emails would have been rejected or sent to us and we still do not receive them or would it be safe to assume that they should be in after a while?

[Michael D.]

thanks. everything is set to echo.supportedns.com already but no emails have been received since we repoted the attack.

 

Hopefully when restored they will all come in at once.

 

Should we trace back with clients in case emails would have been rejected or sent to us and we still do not receive them or would it be safe to assume that they should be in after a while?

I sent you a PM just now.

[iamakio]

Hi Mike and Staffs,

 

I you guys can perform this one quickly. Thanks for informing me, goodluck.

[omaranayat]

Hey Mike,

Please get the accounts moved ASAP. Different time zone here means the attack hit me at the worst time possible. I'm losing valuable business.

 

Good luck!

[Michael D.]

Please get the accounts moved ASAP.

Obviously.

 

Different time zone here means the attack hit me at the worst time possible.

There is never a good time for a DDoS attack.

 

I'm losing valuable business.

If downtime due to issues outside of our control is a serious issue for you I would advise you get with billing and look into moving to a semi-dedicated level account. Semi-dedicated servers have only 1% to 3% of the number of customers that we put on the same hardware for regular shared which means two things:

• We can provide you more CPU resources.
• You are much less likely to be affected by something like a DDoS due to the reduced account number.

Understand my goal is not to make more money off of you but simply to guide you to a service level that is likely more in-line with your needs. It's up to you whether you choose to upgrade or not [no obligation to do so].

[omaranayat]

My point was just to bring to light the possibility that some of your customers are in different time zones so that your team will deal with this situation with the same level of priority, as they would have if the attack had hit 6 hours later or earlier. It's good to know that this is not the case and your team is dealing with it accordingly.

 

Thank you for the suggestion. I was already considering that.

[omaranayat]

IP changed!

Thank you so much for the quick recovery. Really loving my decision to move to MDDHosting.

[bluemartian]

If you are not using our DNS [generally you would know if this is the case - if you do not know, then you are most likely using our DNS] you will need to update your third party DNS once your IP changes. On the left side under 'statistics' you can see your 'Shared IP Address' [you may need to hit 'expand']. Once you see it change from 173.248.188.110 to something different you would want to update your third party DNS.

 

Hi Mike...

 

Sorry for being dumb about this but what you are saying above has me a bit confused. Are you talking about the nameservers we entered at our domain registrar when you mention "third party DNS?" My CPanel shows a Shared IP number different from what you mention above and yet my sites still do not come up. Not sure if I'm supposed to be doing something at this point or just waiting for propagation?

[Michael D.]

Hi Mike...

 

Sorry for being dumb about this but what you are saying above has me a bit confused. Are you talking about the nameservers we entered at our domain registrar when you mention "third party DNS?" My CPanel shows a Shared IP number different from what you mention above and yet my sites still do not come up. Not sure if I'm supposed to be doing something at this point or just waiting for propagation?

If you set your domain to our nameservers there is nothing for you to do but to wait for your IP to be changed and DNS propagation to happen.

 

Again - if you are not using our nameservers and you are using third party DNS - you would know this and, as such, wouldn't have any questions. If you don't know - then you're most likely not using third party DNS because third party DNS is a technical process that requires a good understanding of DNS.

[Michael D.]

All IP changes have been completed. You will want to clear your browser cache, however, due to DNS propagation/caching it may take a couple of hours for the IP changes to take full effect.

 

Understand that this attack can, and likely will, shift to a new IP [we split the accounts up to multiple IPs]. If this happens we will still have customers affected but a substantially smaller amount after which we will be able to narrow down who is under attack.

 

If you are using our DNS there is nothing for you to do but to wait. If you are using third party DNS you will need to log into your cPanel at http://echo.supportedns.com/cpanel and obtain your new IP address from the statistics bar on the left side under, "Shared IP" and update your third party DNS.

 

If you are using third party DNS - you generally would know it. If you are unsure if you are or not, chances are you are not. If you are using CloudFlare that was not configured via our cPanel you will likely need to log into CloudFlare and update your IP there.

[Leah2]

Hi,

 

Kudo's to MDD for the swift response! My clients IP is changed over. I was able to flush the DNS on my Ipad & phone > so I can view it there with no problem.

 

I am using my laptop on a paid Internet service that will not allow me to ipconfig /flushdns. Unfortunately their website is showing the "defaultwebpage.cgi" on my laptop. So I am thinking that my clients & my clients visitors when faced with the directive of the default page will be very confused. Would it be possible for me to set the A record to 60 for a day or so & then reset it back to 14400 in order to get their browsers to flush their DNS faster. I also took off the .htaccess caching of static objects.

 

The reason I am so concerned is that I got them to switch over to MDD to get away from downtime on a wonky server & whammo a DDOS attack. Not that a DDOS attack is by any means controlable by MDD. I'm sure you all could have done without this mornings firedrill!

 

 

Thanks!

[ubshreenath]

Mike

 

Is the IP change permanent? Or is it a temporary fix.

 

I use your DNS on most of my sites but on one domain that a client owns and uses an IP instead of a NS record, I will have to intimate them to update the IP address on GoDaddy.

 

So checking if this IP change is permanent or transient?

 

Please let me know.

 

Sreenath

[Michael D.]

I am using my laptop on a paid Internet service that will not allow me to ipconfig /flushdns.

Flushing your DNS will only flush it on your local system regardless of your ISP. You can switch to something much more responsive like Google Public DNS if you want.

 

Unfortunately their website is showing the "defaultwebpage.cgi" on my laptop.

Make sure that you're not simply 'reloading' the page because you will reload the URL in the URL bar [i.e. the defaultwebage.cgi].

 

So I am thinking that my clients & my clients visitors when faced with the directive of the default page will be very confused.

There are two things to keep in mind here.

1. They will see this page instead of simply never being able to connect and getting no error. It's better than nothing.

2. If they didn't try visiting the site within the hour or two before the IP changed - they will see the new IP immediately and will be able to load the site just fine [as you were able to do via your phone/iPad.]

 

Would it be possible for me to set the A record to 60 for a day or so & then reset it back to 14400 in order to get their browsers to flush their DNS faster.

Changing it now won't cause it to flush faster - but will cause it not to be cached as long in the future. I generally set mine to ~5 minutes.

 

I also took off the .htaccess caching of static objects.

The content of your account will have no bearing on DNS propagation.

 

The reason I am so concerned is that I got them to switch over to MDD to get away from downtime on a wonky server & whammo a DDOS attack. Not that a DDOS attack is by any means controlable by MDD. I'm sure you all could have done without this mornings firedrill!

Let's just say my eyes are burning from a lack of sleep - not a good day. That said - DDoS happens.

 

It's unfortunately very common as the internet is a hostile place. Look at this map to get an idea:

http://www.digitalattackmap.com

 

Thanks!

[Michael D.]

Is the IP change permanent? Or is it a temporary fix.

Likely permanent. If we do change things back up [such as if the attack shifts] we will post it here.

 

I use your DNS on most of my sites but on one domain that a client owns and uses an IP instead of a NS record, I will have to intimate them to update the IP address on GoDaddy.

Most that do this - do this because of custom DNS records. You may want to make sure the client knows that they can create custon DNS Records on our nameservers via cPanel -> Advanced DNS Zone Editor.

[Leah2]

Let's just say my eyes are burning from a lack of sleep - not a good day. That said - DDoS happens.

 

It's unfortunately very common as the internet is a hostile place. Look at this map to get an idea:

http://www.digitalattackmap.com

 

Thanks!

 

Hi Mike,

 

That's one of the main reasons I always recommend MDD. Stuff happens... it's how you handle the "fix" that counts!

 

Thanks for taking the time to explain. I wasn't sure if any of that would help speed up the DNS issue.

 

Thanks again for your help!

[Tim]

 

Hi Mike,

 

That's one of the main reasons I always recommend MDD. Stuff happens... it's how you handle the "fix" that counts!

 

Thanks for taking the time to explain. I wasn't sure if any of that would help speed up the DNS issue.

 

Thanks again for your help!

 

Our pleasure, we appriciate your reccomendations!

[bluemartian]

If you set your domain to our nameservers there is nothing for you to do but to wait for your IP to be changed and DNS propagation to happen.

 

Again - if you are not using our nameservers and you are using third party DNS - you would know this and, as such, wouldn't have any questions. If you don't know - then you're most likely not using third party DNS because third party DNS is a technical process that requires a good understanding of DNS.

 

 

Thanks for the info, Mike. I am familiar with what I call "third party DNS" - Google Public DNS and OpenDNS, both of which I have used in the past, although now I am currently using Level3 as I discovered that OpenDNS takes forever to update. My sites have been back up for a good while now, so I'm assuming you are talking about something else when you refer to "third party DNS." Whatever it is, I much appreciate all your hard work! Thanks.

[Michael D.]

Thanks for the info, Mike. I am familiar with what I call "third party DNS" - Google Public DNS and OpenDNS, both of which I have used in the past, although now I am currently using Level3 as I discovered that OpenDNS takes forever to update.

Not what I meant - I am not talking about for your resolvers. I am talking about for your domains.

 

It's one of two things:

1. You updated your nameservers at your domain registrar to point to our nameservers. [Likely]

2. You are using your registrar's nameservers or a third party DNS service like DNSMadeEasy and have configured all of your DNS zones manually. [unlikely]

 

My sites have been back up for a good while now, so I'm assuming you are talking about something else when you refer to "third party DNS." Whatever it is, I much appreciate all your hard work! Thanks.

Aye, sorry I'm running on no sleep so what I have in my head isn't necessarily making it to the forums the way I intended it.

[Michael D.]

The attack has shifted to the IP '173.248.187.21'.

 

Some customers will be offline - we're working on it.

Correction - it shifted to '173.248.188.167'

 

[Michael D.]

I've gotten with our facility and they believe they can filter this attack completely but it may take ~60 minutes to get the filtration in place. Until this happens the sites on '173.248.188.167' will remain offline to maintain server and network stability. If we bring this IP back online prior to the filtration it will take the entire server [and likely our entire network] offline.

 

Understand if this attack grows substantially the filtration will not be sufficient [i.e. at some point it will begin to degrade the entire network] - if this happens those on the affected IP would remain offline but we're hoping not to get to that point.

[Michael D.]

The filtration is almost in place, ETA ~10 minutes.

 

We're changing some of our routing as well to route our traffic exclusively through the 10GBPS links so that, with luck, it won't overwhelm any links and we can keep everybody online through the attack.