18 replies to this topic

[Scott]

Around 12:30 EDT today, a DDoS attack against a busy site on Hermes began impacting performance across the server on client websites and cPanel. This resulted in some dropped connections as well as some pages taking longer to load than normal. Our staff immediately responded to investigate and mitigate the issue.

The attack has currently been migitated and we are waiting to see if the attack shifts in any way or just stops. At this time, we are not releasing additional details concerning the DDoS attack to help ensure those details are not used against this server.

If you have any quesstions, feel free to ask here or to open a support ticket directly.

More information concerning DDoS attacks can be read here: http://en.wikipedia....-service_attack and here: http://searchsecurit...-service-attack

=============

Update
Sept 9, 2012. About 6:05PM EDT - The attack has returned and we are actively mitigating it.

Edited by MikeDVB, 28 November 2012 - 05:45 PM.
DDoS attack has returned

[MikeDVB]

We had to make some changes to the kernel (operating system) TCP/IP stack to better cope with the amount of packets it's seeing under this attack. Things are, at this point, stable but that isn't to say that can't change if the attackers adapt/change the attack.

Another thing to note, is that it's not us that's under attack - but a customer of ours. Unfortunately the nature of the internet is that it's a very hostile place - we are going to do what we can to keep everybody online and operational.

[MikeDVB]

The attackers have shifted to hitting the MySQL server powering the site that was under attack. We've blocked off remote access to the MySQL server.

[Scott]

We are rebooting the server. It should be back online in about 5 minutes unless an fsck is required.

[karrinina]

Quick question: sometimes in the past, you've moved sites to a new IP address during one of these attacks. Think that's likely with Hermes today? I want to make sure we are standing by to handle updating at one of our site's external name server, if so. Thanks!

[MikeDVB]

You won't be moved as we're not null-routing any IPs.

The server was rebooted, however, MySQL was still being hammered. We've upgraded MySQL from 5.1.52 to 5.5.25 and will be rebuilding PHP modules to match. At this point everything appears to be back to normal, although some scripts may report issues until the PHP modules are rebuilt (takes about an hour).

[MikeDVB]

Everything should be pretty close to normal at this point. You may still see some slowness/intermittent packet loss if you're pinging but we're doing our best to keep things fast and stable. MySQL was upgraded which resolved the MySQL denial of service attack the server was facing as well.

If you have any further issues, by all means feel free to open a ticket or to update any ticket you may already have on the issue.

[SarisIsop]

Just want to say thanks to Mike and Scott for working on this whilst still dealing with me via a ticket. This forum doesn't show all what goes on behind the scenes, but I can assure anyone reading this that MDD always respond quickly to support tickets, and they will work with you to resolve your questions.

[Scott]

I am marking this resolved. It looks like we overlooked doing this earlier.

[MikeDVB]

It appears these attacks have returned, we're working on mitigating them.

[SarisIsop]

I was just about to let you know, I'm off-line again.

[MikeDVB]

The attack is hitting the IP 173.248.187.238 and is of large enough scale that we have no choice at this time but to null-route the IP. We are going to work to move any affected accounts to new IP addresses to hopefully bring them back online.

[MikeDVB]

For any curious:

Keep in mind that 235 MBPS is a lot, but what is more important is the packets per second as well as the type of packets (i.e. UDP, TCP to port 80, actual HTTP requests versus garbage traffic, etc).

The attack is on the order of 1,000,000 (1 million) packets per second.

[MikeDVB]

Null-Route in place, you'll see the blue line drops back to normal right at the right side of the graph:

[SarisIsop]

I'm back on-line, thanks.

[MikeDVB]

Yep .

[joshualoy]

What action do you take on a customer who has a site under attack? I know some providers will instantly terminate the account.

[Scott]

What action do you take on a customer who has a site under attack? I know some providers will instantly terminate the account.

Our first priority is always restoring normal service to other clients on the server by migitating the attack. In terms of how we handle the targeted account, it is evaluated on a case by case basis.

[MikeDVB]

Generally we do our best to keep the target online as well - but we obviously can't make promises in that regard.

We do have some agreements with third party proxy providers that can filter some DDoS attacks (depending on type/size/duration) that we can make available to customers in need at reduced pricing. We've been able to avoid that except for a single case so far this year.