Michael D. Posted November 3, 2011 Report Share Posted November 3, 2011 An IP address on the Fresco server has come under a very large attack (2 GBPS+ and 24 million+ packets per second) and we were forced to null-route the IP to preserve our network and speed for everybody else not on the affected IP address. We are currently investigating to try and identify the target of the attack so that we can safely bring everybody else affected by this null-route back online as soon as possible. If you have any questions, feel free to ask them, however we may not be able to reveal certain details of the attack publicly and I may respond to you via PM with specifics after addressing your question generally here in this thread. Quote Link to comment Share on other sites More sharing options...
Zylantex Posted November 3, 2011 Report Share Posted November 3, 2011 Keep up the good work Mike. We all appreciate it. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted November 3, 2011 Author Report Share Posted November 3, 2011 (edited) Upon closer investigation the attack was closer to 2 GBPS total and 22 to 24 million packets per second. We are moving sites off of the affected IP to different IPs to bring people back online as well as watching for the attack to shift to identify the target. The null-route will be in effect until the attack subsides or the direct target is identified. Unfortunately the IP that was hit, was a shared IP address with multiple clients and it was a flood that targeted the IP and didn't reveal any specific domain as it's target which makes the work more time consuming and difficult. Edited November 3, 2011 by MikeDVB Updated original post with more accurate details. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted November 3, 2011 Author Report Share Posted November 3, 2011 The attack did shift targets with some account moves so we're still working to identify the targeted account. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted November 3, 2011 Author Report Share Posted November 3, 2011 We have applied a dedicated IP to each account that was on the new IP that came under attack, and once the attack moves (it will likely take 1 to 4 hours) we'll know exactly which customer is under attack and will contact them at that point to discuss their options. For now the IP under attack is null-routed until DNS updates for the world for the accounts that were moved, and then the attack will shift again for the last time. This means we will likely face another 2 to 5 minutes of network issues sometime today. We are standing by and monitoring the servers and traffic for this attack shift so that we can quickly take the necessary actions to ensure our network integrity. If you have any questions at all, let us know. Quote Link to comment Share on other sites More sharing options...
008Rohit Posted November 3, 2011 Report Share Posted November 3, 2011 We have applied a dedicated IP to each account that was on the new IP that came under attack, and once the attack moves (it will likely take 1 to 4 hours) we'll know exactly which customer is under attack and will contact them at that point to discuss their options. For now the IP under attack is null-routed until DNS updates for the world for the accounts that were moved, and then the attack will shift again for the last time. This means we will likely face another 2 to 5 minutes of network issues sometime today. We are standing by and monitoring the servers and traffic for this attack shift so that we can quickly take the necessary actions to ensure our network integrity. If you have any questions at all, let us know.I appreciate the information! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted November 3, 2011 Author Report Share Posted November 3, 2011 I appreciate the information!Absolutely. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted November 3, 2011 Author Report Share Posted November 3, 2011 We've identified the target and isolated them, however, the attack is back on an older IP (likely delayed DNS updates) so we had to null route it again and will spot check as possible. Quote Link to comment Share on other sites More sharing options...
fshagan Posted November 3, 2011 Report Share Posted November 3, 2011 Mike, would this have had any impact on the other servers in the data center? I don't think so, but I'm investigating a slowdown on my VPS on Atlantis this morning (I suspect it has to do with the virus / malware scanning I'm doing, but wanted to make sure before I start tweaking things again). Quote Link to comment Share on other sites More sharing options...
Michael D. Posted November 3, 2011 Author Report Share Posted November 3, 2011 Yes, it caused some network wide issues. Quote Link to comment Share on other sites More sharing options...
fshagan Posted November 3, 2011 Report Share Posted November 3, 2011 Thanks, that reassures me about the issue I was seeing this AM. It wasn't really bad, but was a slow down I couldn't resolve. I guess that many packets coming through the pipe affects everyone. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted November 4, 2011 Author Report Share Posted November 4, 2011 Thanks, that reassures me about the issue I was seeing this AM. It wasn't really bad, but was a slow down I couldn't resolve. I guess that many packets coming through the pipe affects everyone.The networking hardware itself can handle around 90 million packets per second if I'm not mistaken but it's only gigabit right now (so the pipe just got flooded). We're looking at going to a 10 GBPS core Q1 2012 and then running probably dual redundant 10 GBPS links to each cabinet and then distributing that to the servers via a 24 port 1 GBPS switch for public networking. Right now it's 1 GBPS end to end which is fine as we average 100 MBPS across our entire network . Quote Link to comment Share on other sites More sharing options...
fshagan Posted November 4, 2011 Report Share Posted November 4, 2011 Wow! The network is certainly fast right now; I never get any complaints about it from any of my clients. But fatter pipes are better. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted November 4, 2011 Author Report Share Posted November 4, 2011 Wow! The network is certainly fast right now; I never get any complaints about it from any of my clients. But fatter pipes are better.The width of the pipe has no bearing on speed unless the pipe gets full which only happens during an extremely large DDoS attack. We're upgrading the core/network in Q1 for a new project that I can't really reveal anything publicly about just yet, but we will be needing a lot more bandwidth so we don't want to put undue stress on our network or cause issues. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted November 9, 2011 Author Report Share Posted November 9, 2011 We did a spot-check on the customer who was the target of the attack and the attack is still ongoing, unfortunately. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.