Jump to content


Photo

[Resolved] 2 GBPS+ DDoS on Fresco Server - Attack affecting entire network intermittenly.


  • Please log in to reply
14 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,200 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 November 2011 - 03:07 AM

An IP address on the Fresco server has come under a very large attack (2 GBPS+ and 24 million+ packets per second) and we were forced to null-route the IP to preserve our network and speed for everybody else not on the affected IP address. We are currently investigating to try and identify the target of the attack so that we can safely bring everybody else affected by this null-route back online as soon as possible.

If you have any questions, feel free to ask them, however we may not be able to reveal certain details of the attack publicly and I may respond to you via PM with specifics after addressing your question generally here in this thread.
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Semi-Dedicated, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/ - Follow us on Twitter!


#2 Zylantex

Zylantex

    Newbie

  • Members
  • Pip
  • 10 posts
  • Gender:Male
  • Location:France

Posted 03 November 2011 - 03:32 AM

Keep up the good work Mike. We all appreciate it.
  • 0

#3 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,200 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 November 2011 - 03:56 AM

Upon closer investigation the attack was closer to 2 GBPS total and 22 to 24 million packets per second. We are moving sites off of the affected IP to different IPs to bring people back online as well as watching for the attack to shift to identify the target. The null-route will be in effect until the attack subsides or the direct target is identified. Unfortunately the IP that was hit, was a shared IP address with multiple clients and it was a flood that targeted the IP and didn't reveal any specific domain as it's target which makes the work more time consuming and difficult.

Edited by MikeDVB, 03 November 2011 - 03:26 PM.
Updated original post with more accurate details.

  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Semi-Dedicated, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/ - Follow us on Twitter!


#4 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,200 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 November 2011 - 07:21 AM

The attack did shift targets with some account moves so we're still working to identify the targeted account.
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Semi-Dedicated, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/ - Follow us on Twitter!


#5 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,200 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 November 2011 - 07:41 AM

We have applied a dedicated IP to each account that was on the new IP that came under attack, and once the attack moves (it will likely take 1 to 4 hours) we'll know exactly which customer is under attack and will contact them at that point to discuss their options. For now the IP under attack is null-routed until DNS updates for the world for the accounts that were moved, and then the attack will shift again for the last time. This means we will likely face another 2 to 5 minutes of network issues sometime today. We are standing by and monitoring the servers and traffic for this attack shift so that we can quickly take the necessary actions to ensure our network integrity.

If you have any questions at all, let us know.
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Semi-Dedicated, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/ - Follow us on Twitter!


#6 008Rohit

008Rohit

    Member

  • Members
  • PipPip
  • 29 posts
  • Gender:Male
  • Location:Kolkata, India
  • Interests:Technology, Smartphones, Web Hosting, Science, Coding.

Posted 03 November 2011 - 07:52 AM

We have applied a dedicated IP to each account that was on the new IP that came under attack, and once the attack moves (it will likely take 1 to 4 hours) we'll know exactly which customer is under attack and will contact them at that point to discuss their options. For now the IP under attack is null-routed until DNS updates for the world for the accounts that were moved, and then the attack will shift again for the last time. This means we will likely face another 2 to 5 minutes of network issues sometime today. We are standing by and monitoring the servers and traffic for this attack shift so that we can quickly take the necessary actions to ensure our network integrity.

If you have any questions at all, let us know.

I appreciate the information!
  • 0

TechTage - Hosted on Gemini Server

Genuine Web Hosting Reviews - Read my MDDHosting Review


#7 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,200 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 November 2011 - 07:53 AM

I appreciate the information!

Absolutely.
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Semi-Dedicated, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/ - Follow us on Twitter!


#8 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,200 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 November 2011 - 09:58 AM

We've identified the target and isolated them, however, the attack is back on an older IP (likely delayed DNS updates) so we had to null route it again and will spot check as possible.
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Semi-Dedicated, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/ - Follow us on Twitter!


#9 fshagan

fshagan

    Member

  • Members
  • PipPip
  • 145 posts

Posted 03 November 2011 - 04:00 PM

Mike, would this have had any impact on the other servers in the data center? I don't think so, but I'm investigating a slowdown on my VPS on Atlantis this morning (I suspect it has to do with the virus / malware scanning I'm doing, but wanted to make sure before I start tweaking things again).
  • 0

#10 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,200 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 November 2011 - 04:09 PM

Yes, it caused some network wide issues.
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Semi-Dedicated, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/ - Follow us on Twitter!


#11 fshagan

fshagan

    Member

  • Members
  • PipPip
  • 145 posts

Posted 03 November 2011 - 06:58 PM

Thanks, that reassures me about the issue I was seeing this AM. It wasn't really bad, but was a slow down I couldn't resolve. I guess that many packets coming through the pipe affects everyone.
  • 0

#12 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,200 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 November 2011 - 09:11 PM

Thanks, that reassures me about the issue I was seeing this AM. It wasn't really bad, but was a slow down I couldn't resolve. I guess that many packets coming through the pipe affects everyone.

The networking hardware itself can handle around 90 million packets per second if I'm not mistaken but it's only gigabit right now (so the pipe just got flooded).

We're looking at going to a 10 GBPS core Q1 2012 and then running probably dual redundant 10 GBPS links to each cabinet and then distributing that to the servers via a 24 port 1 GBPS switch for public networking. Right now it's 1 GBPS end to end which is fine as we average 100 MBPS across our entire network :).
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Semi-Dedicated, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/ - Follow us on Twitter!


#13 fshagan

fshagan

    Member

  • Members
  • PipPip
  • 145 posts

Posted 04 November 2011 - 08:53 AM

Wow! The network is certainly fast right now; I never get any complaints about it from any of my clients. But fatter pipes are better.
  • 0

#14 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,200 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 04 November 2011 - 01:22 PM

Wow! The network is certainly fast right now; I never get any complaints about it from any of my clients. But fatter pipes are better.

The width of the pipe has no bearing on speed unless the pipe gets full which only happens during an extremely large DDoS attack.

We're upgrading the core/network in Q1 for a new project that I can't really reveal anything publicly about just yet, but we will be needing a lot more bandwidth so we don't want to put undue stress on our network or cause issues.
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Semi-Dedicated, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/ - Follow us on Twitter!


#15 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,200 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 09 November 2011 - 02:55 PM

We did a spot-check on the customer who was the target of the attack and the attack is still ongoing, unfortunately.
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Semi-Dedicated, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/ - Follow us on Twitter!





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users