Scott Posted August 31, 2012 Report Share Posted August 31, 2012 (edited) Around 12:30 EDT today, a DDoS attack against a busy site on Hermes began impacting performance across the server on client websites and cPanel. This resulted in some dropped connections as well as some pages taking longer to load than normal. Our staff immediately responded to investigate and mitigate the issue. The attack has currently been migitated and we are waiting to see if the attack shifts in any way or just stops. At this time, we are not releasing additional details concerning the DDoS attack to help ensure those details are not used against this server. If you have any quesstions, feel free to ask here or to open a support ticket directly. More information concerning DDoS attacks can be read here: http://en.wikipedia....-service_attack and here: http://searchsecurit...-service-attack ============= UpdateSept 9, 2012. About 6:05PM EDT - The attack has returned and we are actively mitigating it. Edited November 28, 2012 by MikeDVB DDoS attack has returned Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 31, 2012 Report Share Posted August 31, 2012 We had to make some changes to the kernel (operating system) TCP/IP stack to better cope with the amount of packets it's seeing under this attack. Things are, at this point, stable but that isn't to say that can't change if the attackers adapt/change the attack. Another thing to note, is that it's not us that's under attack - but a customer of ours. Unfortunately the nature of the internet is that it's a very hostile place - we are going to do what we can to keep everybody online and operational. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 31, 2012 Report Share Posted August 31, 2012 The attackers have shifted to hitting the MySQL server powering the site that was under attack. We've blocked off remote access to the MySQL server. Quote Link to comment Share on other sites More sharing options...
Scott Posted August 31, 2012 Author Report Share Posted August 31, 2012 We are rebooting the server. It should be back online in about 5 minutes unless an fsck is required. Quote Link to comment Share on other sites More sharing options...
karrinina Posted August 31, 2012 Report Share Posted August 31, 2012 Quick question: sometimes in the past, you've moved sites to a new IP address during one of these attacks. Think that's likely with Hermes today? I want to make sure we are standing by to handle updating at one of our site's external name server, if so. Thanks! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 31, 2012 Report Share Posted August 31, 2012 You won't be moved as we're not null-routing any IPs. The server was rebooted, however, MySQL was still being hammered. We've upgraded MySQL from 5.1.52 to 5.5.25 and will be rebuilding PHP modules to match. At this point everything appears to be back to normal, although some scripts may report issues until the PHP modules are rebuilt (takes about an hour). Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 31, 2012 Report Share Posted August 31, 2012 Everything should be pretty close to normal at this point. You may still see some slowness/intermittent packet loss if you're pinging but we're doing our best to keep things fast and stable. MySQL was upgraded which resolved the MySQL denial of service attack the server was facing as well. If you have any further issues, by all means feel free to open a ticket or to update any ticket you may already have on the issue. Quote Link to comment Share on other sites More sharing options...
SarisIsop Posted September 1, 2012 Report Share Posted September 1, 2012 Just want to say thanks to Mike and Scott for working on this whilst still dealing with me via a ticket. This forum doesn't show all what goes on behind the scenes, but I can assure anyone reading this that MDD always respond quickly to support tickets, and they will work with you to resolve your questions. Quote Link to comment Share on other sites More sharing options...
Scott Posted September 7, 2012 Author Report Share Posted September 7, 2012 I am marking this resolved. It looks like we overlooked doing this earlier. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 9, 2012 Report Share Posted September 9, 2012 It appears these attacks have returned, we're working on mitigating them. Quote Link to comment Share on other sites More sharing options...
SarisIsop Posted September 9, 2012 Report Share Posted September 9, 2012 I was just about to let you know, I'm off-line again. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 9, 2012 Report Share Posted September 9, 2012 The attack is hitting the IP 173.248.187.238 and is of large enough scale that we have no choice at this time but to null-route the IP. We are going to work to move any affected accounts to new IP addresses to hopefully bring them back online. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 9, 2012 Report Share Posted September 9, 2012 For any curious:http://www.screen-shot.net/2012-09-09_1834.pngKeep in mind that 235 MBPS is a lot, but what is more important is the packets per second as well as the type of packets (i.e. UDP, TCP to port 80, actual HTTP requests versus garbage traffic, etc). The attack is on the order of 1,000,000 (1 million) packets per second. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 9, 2012 Report Share Posted September 9, 2012 Null-Route in place, you'll see the blue line drops back to normal right at the right side of the graph:http://www.screen-shot.net/2012-09-09_1840.png Quote Link to comment Share on other sites More sharing options...
SarisIsop Posted September 9, 2012 Report Share Posted September 9, 2012 I'm back on-line, thanks. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 9, 2012 Report Share Posted September 9, 2012 Yep . Quote Link to comment Share on other sites More sharing options...
joshualoy Posted September 11, 2012 Report Share Posted September 11, 2012 What action do you take on a customer who has a site under attack? I know some providers will instantly terminate the account. Quote Link to comment Share on other sites More sharing options...
Scott Posted September 11, 2012 Author Report Share Posted September 11, 2012 What action do you take on a customer who has a site under attack? I know some providers will instantly terminate the account. Our first priority is always restoring normal service to other clients on the server by migitating the attack. In terms of how we handle the targeted account, it is evaluated on a case by case basis. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 11, 2012 Report Share Posted September 11, 2012 Generally we do our best to keep the target online as well - but we obviously can't make promises in that regard. We do have some agreements with third party proxy providers that can filter some DDoS attacks (depending on type/size/duration) that we can make available to customers in need at reduced pricing. We've been able to avoid that except for a single case so far this year. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.