[Ended] Fresco server - DDoS attack

[Michael D.]

Hello,

 

One of our clients on the Fresco server has come under DDoS attack which has caused connection speeds to be a bit slower than usual for about the last hour due to maxing out the port on the server. We're upgrading the port on the server to 1,000 megabit from the current 100 megabit which only takes moments and we're engaging our Cisco Guard anti-DDoS on the site.

 

We've contacted the client in question so you don't have to ask here if it's you or not (if it is you, then you already know that).

 

If you have any questions at all, feel free to post in this thread or to open a support ticket.

 

Bandwidth Graph:

http://www.screen-shot.net/2010-09-26_1940.png

 

Thank you!

[jrotunda85]

Mike,

 

Is the person that is getting attacked now the same one that was targeted on the previous outage? It would seem to me if they are indeed the same person, their account should be suspended immediately.

[Michael D.]

Mike,

 

Is the person that is getting attacked now the same one that was targeted on the previous outage? It would seem to me if they are indeed the same person, their account should be suspended immediately.

Different server, different client, entirely different type of attack.

 

Unfortunately this is part of the industry - of the dozen or so other providers I speak with on a daily basis at least two of them are dealing with a type of DDoS attack at any given time. We're just letting you know what is happening and what we're doing about it. The server has been put behind CiscoGuard and the client has been contacted so that we can work with them on this issue.

 

If you have any questions, let us know.

[Michael D.]

The attack at this point is being fully mitigated so you shouldn't notice any impact on any services. If you do have any issues or questions feel free to post them in this thread or to open a support ticket.

http://www.screen-shot.net/2010-09-26_2200.png

[Michael D.]

We did some math and the site under attack was facing upwards of 43,200,000 requests per hour, 720,000 per minute, and 12,000 per second. That's a LOT of requests! This type of attack wasn't aimed at flooding our pipes with incoming data but by overworking the web server (i.e. crashing it) or simply causing bandwidth usage to be too high/expensive to keep the site online.

 

We did manage to mitigate a good portion of the attack and things are definitely back to normal. If you do have any questions at all, let us know.

[Michael D.]

The attack is still ongoing, we're seeing upwards of 2,000 distinct systems hitting the server and upwards of 15 to 20 thousand requests per second. Things may be a *little* sluggish but we're doing our best to keep the server responsive.

[Michael D.]

We're having a null-route applied to the IP address that is being hit by the attack and we're moving all sites that aren't targets of the attack (if your site was the target, you were already notified and responded) to new IP addresses.

[dimples]

oh this is why my site so slow..and can't connect sftp

which account or site getting attack?

hope that issues taking care soon..

thank you

[Michael D.]

oh this is why my site so slow..and can't connect sftp

Yes - DDoS attacks tend to have that effect. We've managed to mitigate enough of the attack that at least sites are online (although sluggish) but we're still working to fully mitigate the attack.

which account or site getting attack?

I can't say due to our privacy policies - you can be sure it's not you though as we've already contacted and discussed the issue with the client whose site is the target.

hope that issues taking care soon..

thank you

We're working on it!

[dimples]

thank you mike

[Michael D.]

Here's a copy of the mass email that we sent out to all affected clients (just in case you didn't get the email)

The DDoS attack is still ongoing and is causing a fair amount of packet loss - to the point of sites being sluggish. We're moving any sites that share the same IP address with the site under attack to new IP addresses so that we can null-route the attack and bring performance back to it's normal levels. As with any IP change your site may appear offline for up to 2 hours to anybody who has accessed the site recently (within the last hour or so) however anybody who hasn't been to your site in the last hour or so won't see your site as offline, and this is a majority of the world.

 

If your site is currently on a Dedicated IP that means that it's on an IP all it's own, and wouldn't be affected by the IP changes. If you do have any questions at all feel free to respond to this email or to post in the topic on our forums about this issue at http://forums.mddhosting.com/topic/322-mitigated-fresco-server-ddos-attack/

 

As always, we're doing our best to keep things online through this attack. The attack is not targeting MDDHosting, but instead one of our customers - we've already contacted the customer who is the target of the attack and discussed the details with them so you don't have to worry about whether you're the target or not, as you would already know.

[alans73]

I was wondering why our site was sluggish and eventually went offline...thanks for the info

[Michael D.]

I was wondering why our site was sluggish and eventually went offline...thanks for the info

No problem, if anything changes we'll post it here.

[BrickMaster]

Hi there, quick question Mr. Denney. My website is currently offline, am I one of those affected by this?

[Scott]

Hi there, quick question Mr. Denney. My website is currently offline, am I one of those affected by this?

 

Please open a support ticket so that we may look into your account directly. Thank you.

[BrickMaster]

Please open a support ticket so that we may look into your account directly. Thank you.

 

All right. Will do sir.

[Scott]

Hi there, quick question Mr. Denney. My website is currently offline, am I one of those affected by this?

 

Some notes for those who are interested:

1. If you're account is not located on the fresco server, then this issue does not apply to you.
2. We have just completed moving all sites on the old fresco IP address to the new IP address. If you're DNS is hosted with us, then it may take a few hours for the changes to propagate to anyone who has visited your site recently.
3. If you're DNS is not hosted with us, you will need to update your DNS records.
4. If you have a dedicated IP, there should be no changes to the IP address of your account and the DNS changes do not apply to you.
5. A few hours from now, if you still notice any problems, please let us know.

[Bluesplayer]

Is this the reason why my traffic has collapsed?

 

http://bluesplayer.co.uk/images/traffic.jpg

 

Do I need to do anything or just wait for the propagation to take effect?

[Michael D.]

Propagation is already completed as of several hours ago, I couldn't speculate about your traffic levels.

[Bluesplayer]

Well it must be linked surely. As you can see the drop in traffic started about 5-6 hours ago. These changes must impact on the search engines really badly. No doubt traffic will recover again but if my site was a viable commercial one earning lots of money I don't think I would be too happy about it. There would be considerable loss of earnings.

 

What exactly is a DDos attack?

[Michael D.]

http://en.wikipedia.org/wiki/Denial-of-service_attack

 

I understand the frustration - I've been personally working on mitigating this attack for over 24 solid hours.

[Bluesplayer]

Ok I understand what a ddos attack is. Just got up only to find my site has practically zero traffic which is very annoying after it has recovered from my own imposed site disruptions (caused by me I would add).

 

I am just wondering why a site under attack can't be switched off so that the rest of the server isn't affected? Why can't such a site then be moved to a different server or something so that any changes being made only affect that one particular site?

[Mike]

I am just wondering why a site under attack can't be switched off so that the rest of the server isn't affected? Why can't such a site then be moved to a different server or something so that any changes being made only affect that one particular site?

One very simple reason is when a site comes under this kind of attack, it can be extremely difficult to determine what traffic is legitimate from what is malicious. Would you want legitimate requests to your site to be 'switched off', if the attack could be mitigated by other means?

 

One method to assure you have no future problems is to move your account to a dedicated server, then you have completely isolated yourself from any other sites.

 

Mike, I co-admin a site owned by one of your customers and I stopped by to offer my thanks for handling the attack as efficiently you did. The site was a bit lazy for a few hours, but once propagation had taken place, we bounced right back. I understand the situation you were in and I do not envy you at all. Your actions exemplified your level of dedication to your customers and did not go unnoticed.

 

For anyone looking for a new host, MDDHosting takes great care of their customers and I suggest you give them a closer look. It's simple to be a great host when things are running smoothly. Mike proved himself and his customer service in a difficult, trying situation and that says a lot.

[Bluesplayer]

As annoying as my replies can be I still would like my opinion heard.

 

For one thing if my site was the cause of lots of other sites being slowed considerably, or going offline, I would expect it to be switched off immediately while a solution was ongoing. I expect that the owner of the affected site would also.

 

God knows what the impact on search engines is with regard to a different ip but I am now seeing the lowest traffic on my site ever:

 

http://whos.amung.us/stats/4xpxrrw7o8zu/

 

In my opinion the affected site should have been isolated straight away and some other way of combatting the ddos attack found.

 

I think maybe you haven't considered the search engine implications of the changes you have made. Who knows how long my site - and others - will take to recover.

[Michael D.]

As annoying as my replies can be I still would like my opinion heard.

First and foremost your replies are not annoying and we are always happy to hear your opinion even if we don't necessarily agree with it

 

For one thing if my site was the cause of lots of other sites being slowed considerably, or going offline, I would expect it to be switched off immediately while a solution was ongoing. I expect that the owner of the affected site would also.

It all depends on the type of attack - some are easy to block and reroute and some are not. This particular account was being attacked by IP and not by it's domain name so the only way to disable the account and the attack was to null-route the IP which necessitated moving all accounts off of that IP but the one under attack which we did yesterday.

 

God knows what the impact on search engines is with regard to a different ip but I am now seeing the lowest traffic on my site ever:

http://whos.amung.us/stats/4xpxrrw7o8zu/

I understand but keep in mind that with most providers your site would have simply been offline until the attack subsided as you were on the same IP - the attack is still ongoing even now.

 

In my opinion the affected site should have been isolated straight away and some other way of combatting the ddos attack found.

What makes you think it wasn't? It takes time to move accounts around to make what we did possible.

 

I think maybe you haven't considered the search engine implications of the changes you have made. Who knows how long my site - and others - will take to recover.

I think maybe you don't really know what we have and haven't considered - keep in mind that our number one goal is keeping your sites online as much as possible. We always do our best however DDoS attacks are tough to deal with.