Jump to content


Photo

Hermes - DDoS

Resolved

  • Please log in to reply
18 replies to this topic

#1 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 31 August 2012 - 01:44 PM

Around 12:30 EDT today, a DDoS attack against a busy site on Hermes began impacting performance across the server on client websites and cPanel. This resulted in some dropped connections as well as some pages taking longer to load than normal. Our staff immediately responded to investigate and mitigate the issue.

The attack has currently been migitated and we are waiting to see if the attack shifts in any way or just stops. At this time, we are not releasing additional details concerning the DDoS attack to help ensure those details are not used against this server.

If you have any quesstions, feel free to ask here or to open a support ticket directly.

More information concerning DDoS attacks can be read here: http://en.wikipedia....-service_attack and here: http://searchsecurit...-service-attack

=============

Update
Sept 9, 2012. About 6:05PM EDT - The attack has returned and we are actively mitigating it.

Edited by MikeDVB, 28 November 2012 - 05:45 PM.
DDoS attack has returned

  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#2 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 31 August 2012 - 02:08 PM

We had to make some changes to the kernel (operating system) TCP/IP stack to better cope with the amount of packets it's seeing under this attack. Things are, at this point, stable but that isn't to say that can't change if the attackers adapt/change the attack.

Another thing to note, is that it's not us that's under attack - but a customer of ours. Unfortunately the nature of the internet is that it's a very hostile place - we are going to do what we can to keep everybody online and operational.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#3 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 31 August 2012 - 02:50 PM

The attackers have shifted to hitting the MySQL server powering the site that was under attack. We've blocked off remote access to the MySQL server.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#4 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 31 August 2012 - 03:10 PM

We are rebooting the server. It should be back online in about 5 minutes unless an fsck is required.
  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#5 karrinina

karrinina

    Newbie

  • Clients
  • Pip
  • 4 posts

Posted 31 August 2012 - 03:25 PM

Quick question: sometimes in the past, you've moved sites to a new IP address during one of these attacks. Think that's likely with Hermes today? I want to make sure we are standing by to handle updating at one of our site's external name server, if so. Thanks!
  • 0

#6 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 31 August 2012 - 03:27 PM

You won't be moved as we're not null-routing any IPs.

The server was rebooted, however, MySQL was still being hammered. We've upgraded MySQL from 5.1.52 to 5.5.25 and will be rebuilding PHP modules to match. At this point everything appears to be back to normal, although some scripts may report issues until the PHP modules are rebuilt (takes about an hour).
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#7 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 31 August 2012 - 03:57 PM

Everything should be pretty close to normal at this point. You may still see some slowness/intermittent packet loss if you're pinging but we're doing our best to keep things fast and stable. MySQL was upgraded which resolved the MySQL denial of service attack the server was facing as well.

If you have any further issues, by all means feel free to open a ticket or to update any ticket you may already have on the issue.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#8 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 155 posts
  • Gender:Not Telling

Posted 01 September 2012 - 05:14 AM

Just want to say thanks to Mike and Scott for working on this whilst still dealing with me via a ticket. This forum doesn't show all what goes on behind the scenes, but I can assure anyone reading this that MDD always respond quickly to support tickets, and they will work with you to resolve your questions.

:)
  • 0

#9 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 07 September 2012 - 08:12 AM

I am marking this resolved. It looks like we overlooked doing this earlier.
  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#10 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 09 September 2012 - 05:07 PM

It appears these attacks have returned, we're working on mitigating them.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#11 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 155 posts
  • Gender:Not Telling

Posted 09 September 2012 - 05:10 PM

I was just about to let you know, I'm off-line again.
  • 0

#12 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 09 September 2012 - 05:26 PM

The attack is hitting the IP 173.248.187.238 and is of large enough scale that we have no choice at this time but to null-route the IP. We are going to work to move any affected accounts to new IP addresses to hopefully bring them back online.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#13 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 09 September 2012 - 05:35 PM

For any curious:
Posted Image
Keep in mind that 235 MBPS is a lot, but what is more important is the packets per second as well as the type of packets (i.e. UDP, TCP to port 80, actual HTTP requests versus garbage traffic, etc).

The attack is on the order of 1,000,000 (1 million) packets per second.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#14 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 09 September 2012 - 05:40 PM

Null-Route in place, you'll see the blue line drops back to normal right at the right side of the graph:
Posted Image
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#15 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 155 posts
  • Gender:Not Telling

Posted 09 September 2012 - 06:01 PM

I'm back on-line, thanks.
  • 0

#16 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 09 September 2012 - 06:03 PM

Yep :).
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#17 joshualoy

joshualoy

    Newbie

  • Members
  • Pip
  • 19 posts

Posted 10 September 2012 - 10:40 PM

What action do you take on a customer who has a site under attack? I know some providers will instantly terminate the account.
  • 0

#18 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 10 September 2012 - 10:44 PM

What action do you take on a customer who has a site under attack? I know some providers will instantly terminate the account.


Our first priority is always restoring normal service to other clients on the server by migitating the attack. In terms of how we handle the targeted account, it is evaluated on a case by case basis.
  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#19 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 10 September 2012 - 11:09 PM

Generally we do our best to keep the target online as well - but we obviously can't make promises in that regard.

We do have some agreements with third party proxy providers that can filter some DDoS attacks (depending on type/size/duration) that we can make available to customers in need at reduced pricing. We've been able to avoid that except for a single case so far this year.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users