Scott Posted March 12, 2011 Report Share Posted March 12, 2011 Update #2 (7:00PM EST):The DDoS attack on Cypress is now resolved.Everything on the Cypress server should be returning to normal. We have null routed the affected IP addresses and moved all clients to a new shared IP. If you are still having issues, try clearing your DNS cache and then open a support ticket so we can look into your account specifically. If you are the target of the attack, we will open a ticket with you and let you know. If you didn't see any such ticket, then the attack wasn't targeting you The remainder of this post will remain unchanged to keep a full record of events intact. ==================== Update #1:For those of you just joining us: We are seeing a 1,300 mbps DDoS attack aimed at our Cypress server. We are currently working with our datacenters networking team to null route affected IPs and migrate clients on those IP's to a new one. If you are using external DNS, such as CloudFlare, and you do NOT have a dedicated IP, please open a ticket so that we may get the new IP address to you when it is available. The remainder of this post will remain how it was originally to keep a full record of events intact. ==================== We are aware of a current issue with the Cypress server and we are currently investigating the cause of this issue. More information will be made available as soon as we have it. Link to comment Share on other sites More sharing options...
Michael D. Posted March 12, 2011 Report Share Posted March 12, 2011 It looks as though a rather large packet flood hit the web server just now, the attack size is not terribly large however with the amount of requests that hit the server it wasn't able to keep up and ended up choking out. We're working on identifying the attack, mitigating it, and we're going to do what we can to prevent it from happening again however the internet is a hostile environment and attacks are inevitable. Link to comment Share on other sites More sharing options...
Michael D. Posted March 12, 2011 Report Share Posted March 12, 2011 http://www.screen-shot.net/2011-03-12_1725.png It seems that today's issue and yesterday's issue are similar and likely related. It's a large influx of traffic into the server that causes the connection tracking in the server to fail resulting in the server becoming unresponsive. We're still working on identifying the source/destination and investigating ways to prevent this from happening again. The server is online however it does take 10 to 15 minutes for it to "catch up" after being rebooted. Link to comment Share on other sites More sharing options...
Brian Stevenson Posted March 12, 2011 Report Share Posted March 12, 2011 http://www.screen-shot.net/2011-03-12_1725.png It seems that today's issue and yesterday's issue are similar and likely related. It's a large influx of traffic into the server that causes the connection tracking in the server to fail resulting in the server becoming unresponsive. We're still working on identifying the source/destination and investigating ways to prevent this from happening again. The server is online however it does take 10 to 15 minutes for it to "catch up" after being rebooted.Thanks for the update. Link to comment Share on other sites More sharing options...
TotalZen Posted March 12, 2011 Report Share Posted March 12, 2011 Yeah thanks for being right on top of this Mike Link to comment Share on other sites More sharing options...
Michael D. Posted March 12, 2011 Report Share Posted March 12, 2011 We just got with the network operations team at our facility and they're definitely seeing an ingress 500 MBPS+ hitting the server. We're putting a null-route in place for the target at which point the server will come back online. The IP that is going to be null-routed does have several customers on it and we will be migrating them individually to new IPs so that we can identify the target of the attack more specifically and then that one particular account will be null routed until the attack subsides. If you have any questions we do ask that you try and ask them here unless they are account specific so that we can provide centralized disbursement of information. Link to comment Share on other sites More sharing options...
Michael D. Posted March 12, 2011 Report Share Posted March 12, 2011 The attack is in excess of 1.3 GBPS / 1,300 MBPS at last check and we're still working with the networking team. Link to comment Share on other sites More sharing options...
Brian Stevenson Posted March 12, 2011 Report Share Posted March 12, 2011 We just got with the network operations team at our facility and they're definitely seeing an ingress 500 MBPS+ hitting the server. We're putting a null-route in place for the target at which point the server will come back online. The IP that is going to be null-routed does have several customers on it and we will be migrating them individually to new IPs so that we can identify the target of the attack more specifically and then that one particular account will be null routed until the attack subsides. If you have any questions we do ask that you try and ask them here unless they are account specific so that we can provide centralized disbursement of information.Is the new ip address temporary or permanent? I'll need to update my nameserver as I have my DNS hosted at cloudflare. Peace,Brian Link to comment Share on other sites More sharing options...
Michael D. Posted March 12, 2011 Report Share Posted March 12, 2011 It would be temporary, if you're on a dedicated IP already then your IP will not be changed however if you're on a shared IP then there is a good chance it will change. Go ahead and open a ticket Brian and I'll get the new IP to you there. Link to comment Share on other sites More sharing options...
Michael D. Posted March 12, 2011 Report Share Posted March 12, 2011 All sites on the affected IP were migrated to new IP addresses. Do feel free to open a ticket if you're not using our DNS however if you are using our DNS you won't need to do anything. Link to comment Share on other sites More sharing options...
Michael D. Posted March 12, 2011 Report Share Posted March 12, 2011 As we thought, the attack followed the DNS change so we're initiating a reboot to get the server back to a state where it's not overwhelmed and we can identify the target IP and then null route that specific IP so that everybody else is online without issues. We hope to have this resolved within the next 10 to 15 minutes. Link to comment Share on other sites More sharing options...
Lincoln Posted March 12, 2011 Report Share Posted March 12, 2011 How does the number of DDOS attacks Cypress experiences compare to the other servers? It seems like its always Cypress that's getting attacked all the time? Link to comment Share on other sites More sharing options...
Scott Posted March 12, 2011 Author Report Share Posted March 12, 2011 How does the number of DDOS attacks Cypress experiences compare to the other servers? It seems like its always Cypress that's getting attacked all the time? I'm not aware of the statistics on this off hand. We do post a public record of events such as this on these forums, so you should be able to look through the history and see when other servers got hit. Once this issue is resolved, we may be able to spend more time looking into the history of such things. Link to comment Share on other sites More sharing options...
TotalZen Posted March 12, 2011 Report Share Posted March 12, 2011 How does the number of DDOS attacks Cypress experiences compare to the other servers? It seems like its always Cypress that's getting attacked all the time? It's probably the same site that's being targeted repeatedly, I would guess. Link to comment Share on other sites More sharing options...
Michael D. Posted March 12, 2011 Report Share Posted March 12, 2011 How does the number of DDOS attacks Cypress experiences compare to the other servers? It seems like its always Cypress that's getting attacked all the time?To my knowledge Cypress has never been hit with a DDoS we weren't able to quickly mitigate in the past and it has been quite a while since our last really damaging DDoS attack. We do leave all of the threads here on the forum so you're welcome to look into it. It's probably the same site that's being targeted repeatedly, I would guess.No - if you become the target of a DDoS you get moved to a Dedicated IP and you stay there - if you ARE repeatedly hit to the point that it's causing issues for our other customers we would have no choice but to ask you to move to a service where you wouldn't affect others such as a dedicated server. The site that is under attack has just been identified and has never been attacked before in it's 2+ years of being hosted with us. We will be getting in contact with the site operator meanwhile everything should normalize for everybody else within the next few minutes. Link to comment Share on other sites More sharing options...
TotalZen Posted March 12, 2011 Report Share Posted March 12, 2011 No - if you become the target of a DDoS you get moved to a Dedicated IP and you stay there - if you ARE repeatedly hit to the point that it's causing issues for our other customers we would have no choice but to ask you to move to a service where you wouldn't affect others such as a dedicated server. The site that is under attack has just been identified and has never been attacked before in it's 2+ years of being hosted with us. I see, good to know. Link to comment Share on other sites More sharing options...
le.gentleman Posted March 12, 2011 Report Share Posted March 12, 2011 Thanks for working on it. DDoS attacks can happenbut you always recognize the problem right away and work on it. I really appreciate this kind of work ethic. My URL is registered with 1and1 (I know they are far from good but so far I did never have issues with them).Will I have to change the name server as well? Link to comment Share on other sites More sharing options...
Scott Posted March 12, 2011 Author Report Share Posted March 12, 2011 My URL is registered with 1and1 (I know they are far from good but so far I did never have issues with them).Will I have to change the name server as well? If you're using our nameservers, you will not need to change anything. If you're using something other than our nameservers, such as CloudFlare or an external DNS server then yes, you will need to contact our support department for your new IP address. Link to comment Share on other sites More sharing options...
le.gentleman Posted March 12, 2011 Report Share Posted March 12, 2011 Thanks Mike, I am indeed on your nameservers . Link to comment Share on other sites More sharing options...
Michael D. Posted March 12, 2011 Report Share Posted March 12, 2011 Thanks Mike, I am indeed on your nameservers .That was actually Scott that responded to you Beyond that, everybody should be back online as we've moved the site under attack to it's own IP address and null routed that IP. If you are still having issues by all means do open a ticket so we can look specifically into your account. Link to comment Share on other sites More sharing options...
Scott Posted March 13, 2011 Author Report Share Posted March 13, 2011 This topic is now marked as resolved. Please see the first post for more information. Link to comment Share on other sites More sharing options...
Recommended Posts