14 replies to this topic

[MikeDVB]

An IP address on the Fresco server has come under a very large attack (2 GBPS+ and 24 million+ packets per second) and we were forced to null-route the IP to preserve our network and speed for everybody else not on the affected IP address. We are currently investigating to try and identify the target of the attack so that we can safely bring everybody else affected by this null-route back online as soon as possible.

If you have any questions, feel free to ask them, however we may not be able to reveal certain details of the attack publicly and I may respond to you via PM with specifics after addressing your question generally here in this thread.

[Zylantex]

Keep up the good work Mike. We all appreciate it.

[MikeDVB]

Upon closer investigation the attack was closer to 2 GBPS total and 22 to 24 million packets per second. We are moving sites off of the affected IP to different IPs to bring people back online as well as watching for the attack to shift to identify the target. The null-route will be in effect until the attack subsides or the direct target is identified. Unfortunately the IP that was hit, was a shared IP address with multiple clients and it was a flood that targeted the IP and didn't reveal any specific domain as it's target which makes the work more time consuming and difficult.

Edited by MikeDVB, 03 November 2011 - 03:26 PM.
Updated original post with more accurate details.

[MikeDVB]

The attack did shift targets with some account moves so we're still working to identify the targeted account.

[MikeDVB]

We have applied a dedicated IP to each account that was on the new IP that came under attack, and once the attack moves (it will likely take 1 to 4 hours) we'll know exactly which customer is under attack and will contact them at that point to discuss their options. For now the IP under attack is null-routed until DNS updates for the world for the accounts that were moved, and then the attack will shift again for the last time. This means we will likely face another 2 to 5 minutes of network issues sometime today. We are standing by and monitoring the servers and traffic for this attack shift so that we can quickly take the necessary actions to ensure our network integrity.

If you have any questions at all, let us know.

[008Rohit]

We have applied a dedicated IP to each account that was on the new IP that came under attack, and once the attack moves (it will likely take 1 to 4 hours) we'll know exactly which customer is under attack and will contact them at that point to discuss their options. For now the IP under attack is null-routed until DNS updates for the world for the accounts that were moved, and then the attack will shift again for the last time. This means we will likely face another 2 to 5 minutes of network issues sometime today. We are standing by and monitoring the servers and traffic for this attack shift so that we can quickly take the necessary actions to ensure our network integrity.

If you have any questions at all, let us know.

I appreciate the information!

[MikeDVB]

I appreciate the information!

Absolutely.

[MikeDVB]

We've identified the target and isolated them, however, the attack is back on an older IP (likely delayed DNS updates) so we had to null route it again and will spot check as possible.

[fshagan]

Mike, would this have had any impact on the other servers in the data center? I don't think so, but I'm investigating a slowdown on my VPS on Atlantis this morning (I suspect it has to do with the virus / malware scanning I'm doing, but wanted to make sure before I start tweaking things again).

[MikeDVB]

Yes, it caused some network wide issues.

[fshagan]

Thanks, that reassures me about the issue I was seeing this AM. It wasn't really bad, but was a slow down I couldn't resolve. I guess that many packets coming through the pipe affects everyone.

[MikeDVB]

Thanks, that reassures me about the issue I was seeing this AM. It wasn't really bad, but was a slow down I couldn't resolve. I guess that many packets coming through the pipe affects everyone.

The networking hardware itself can handle around 90 million packets per second if I'm not mistaken but it's only gigabit right now (so the pipe just got flooded).

We're looking at going to a 10 GBPS core Q1 2012 and then running probably dual redundant 10 GBPS links to each cabinet and then distributing that to the servers via a 24 port 1 GBPS switch for public networking. Right now it's 1 GBPS end to end which is fine as we average 100 MBPS across our entire network .

[fshagan]

Wow! The network is certainly fast right now; I never get any complaints about it from any of my clients. But fatter pipes are better.

[MikeDVB]

Wow! The network is certainly fast right now; I never get any complaints about it from any of my clients. But fatter pipes are better.

The width of the pipe has no bearing on speed unless the pipe gets full which only happens during an extremely large DDoS attack.

We're upgrading the core/network in Q1 for a new project that I can't really reveal anything publicly about just yet, but we will be needing a lot more bandwidth so we don't want to put undue stress on our network or cause issues.

[MikeDVB]

We did a spot-check on the customer who was the target of the attack and the attack is still ongoing, unfortunately.