Atlantis DDoS Attack - July 26, 2009 (Details + Discussion)

[Michael D.]

Hello.

 

First I would like to apologize for the lack of information this morning - while I am happy that the overnight staff were focusing their efforts on mitigating the incoming attack - I am not so happy with the amount of information that they didn't send to anybody who opened a ticket which is something I am going to put policies in place to address.

 

The Atlantis server came under attack on a Single IP this morning which we began filtering out with Cisco Guard - the attack wasn't large enough to trip the automatic protection but it was large enough to slow things down and cause some connection issues. As soon as the attack began to be filtered out the attacker moved the attack to a new IP on the server.

 

This attacker seemed determined to bring our services offline this morning and I am sad to say that they did a pretty decent job. In the end our datacenter went ahead and applied Cisco Guard to all IP addresses on the server (normally this would be very expensive) since every time we began to mitigate the attack it just jumped IP addresses.

 

We're still investigating the source of the attack although we do not believe this attack to be related to the attack that hit Boreas last week.

 

It is truly sad that people choose to commit such actions.

[Michael D.]

Here is an image of the bandwidth graph for today:

http://www.screen-shot.net/ss/95863184231235784311.png

 

And for the last month:

http://www.screen-shot.net/ss/57234776961144871842.png

 

As you can see the attack was very out of the ordinary traffic.

[Michael D.]

I've written a blog post that is related to this that you may find interesting. You certainly don't have to read it if you don't want to

 

http://www.mikedvb.com/2009/07/26/another-day-another-dos/

 

You are also more than welcome to discuss this particular attack in this thread.

[Michael D.]

The attacker has really stepped things up and their attack is making it past Cisco Guard, our networking department is working as hard as they can to restore service however at this time it seems to be a cat and mouse game as every time we make a change to filter their attack they change the attack.

[patlaw]

What is the attraction to a DOS attack? What is the scumbag hoping to accomplish?

[Michael D.]

What is the attraction to a DOS attack? What is the scumbag hoping to accomplish?

If only I knew.

 

Right now as it stands the attack is making it past CiscoGuard.

 

We are moving our DNS to a new IP and we are going to be shifting all domains over (except the one being attacked) to new IPs and then null-routing that particular IP.

 

If you are using Custom DNS please contact support at support@mddhosting.com.

[Michael D.]

I am writing an update e-mail that will go to everybody hosted on the server.

[Michael D.]

My network administrator has recommend that we redirect all traffic for that domain to another IP that is being null-routed as we speak and then dropping the null-route on the primary IP. Depending on how long DNS takes to update this will offer the fastest resolution to the issue without forcing everybody to modify their DNS.

 

And for the record, you are welcome to respond in this thread and ask any questions you may have.

 

The settings have taken effect and we are seeing that all services are coming back online - if you see that your site is back online by all means post and let us know.

[Michael D.]

As DNS changes the server is still going to be seeing some hits from the DDoS so things are going to be slower than usual however we anticipate within about 2 hours maximum DNS should be fully updated.

 

If you have any issues open a support ticket.

[RDS Inc.]

My network administrator has recommend that we redirect all traffic for that domain to another IP that is being null-routed as we speak and then dropping the null-route on the primary IP. Depending on how long DNS takes to update this will offer the fastest resolution to the issue without forcing everybody to modify their DNS.

 

And for the record, you are welcome to respond in this thread and ask any questions you may have.

 

The settings have taken effect and we are seeing that all services are coming back online - if you see that your site is back online by all means post and let us know.

 

I am reporting that service seems to have returned but it is now abnormally slow (according to the server status page Atlantis seems to have a heavy load.).

[Michael D.]

I am reporting that service seems to have returned but it is now abnormally slow (according to the server status page Atlantis seems to have a heavy load.).

Yeah, some of the attack is still hitting the server.

 

The problem is that this attack simulates actual traffic very well and is coming from nearly 5,000 individual IP addresses so it's next to impossible to filter it out. I would think that DDoS protection such as CiscoGuard would give you the ability to filter traffic destined for a specific domain however it seems that is lacking from the DDoS protection field.

 

What we have done is forced all requests for that particular domain to a new IP via DNS however it is going to take time for that DNS to propagate so as that happens the speed will continue to increase.

 

We have certainly learned quite a bit from this attack as it is the largest and the most devastating that we have ever experienced and we are going to be making some very important changes over the next 72 hours to allow us to better cope with such an attack in the future.

 

While I am happy that we have been able to filter this attack much faster than the average provider - most providers that experience such an attack are down for a week or more; I am still not happy overall with how well we have handled the situation. We've handled the situation better than 99.9% of all other hosting companies out there would however in my eyes it's still not quite good enough which is why I am going to make sure some new policies and procedures are put in place as soon as possible.

 

If you have any questions, you can post them here as we will be watching, or you are welcome to open a support ticket. Keep in mind that support ticket responses may be a tad slower than you are used to however we will get to you as soon as possible.

[Michael D.]

The attacker has changed methods and is hitting our server with approximately between 180,000 and 250,000 requests per second from around 10,000 individual IP addresses.

 

Our Cisco Guard is filtering out a good majority of this attack however even the bit that is getting by is still far more than a server can even handle. I have our best server administrators working to optimize the server for a tremendously large amount of connections (more than you would ever see on a normal server) and our best networking engineers working to attempt to mitigate the attack.

 

This attack is the largest attack I have personally ever seen in my history of the industry and in speaking with other providers that I have networked with it is the largest that they have seen.

 

I can assure you that we are doing our absolute best to resolve this and I can tell you that I personally won't be getting any sleep until all services are fully restored.

 

For anybody using PING or TRACEROUTE to see if the server is online, at this time we are blocking ALL ICMP packets so Ping and Traceroute will fail even though the server is online.

 

Please simply check your site by visiting it in your browser.

[Lightpix]

The attacker has changed methods and is hitting our server with approximately between 180,000 and 250,000 requests per second from around 10,000 individual IP addresses.

 

 

This attack is the largest attack I have personally ever seen in my history of the industry and in speaking with other providers that I have networked with it is the largest that they have seen.

 

It seems that this is a phenomenal attack. Are they attacking a single IP (domain) specifically, which will improve as your DNS change is propagated, and the net effect is to attack the server? Or are they attacking the server broadly, at every domain they can find that is listed under a given IP (does that make sense)?

 

If this question asks for too much info, please feel free to delete or alter.

 

I assume the FBI is involved with the problem as well?

 

Lightpix

[Michael D.]

It seems that this is a phenomenal attack. Are they attacking a single IP (domain) specifically

The attack started out by targeting only a single domain and we quickly re-routed that domain and then null-routed it's new location and the attackers switched from that domain to the general IP of the server.

 

which will improve as your DNS change is propagated,

Things did improve and we had everything back online and then the attacker shifted the attack.

 

and the net effect is to attack the server? Or are they attacking the server broadly, at every domain they can find that is listed under a given IP (does that make sense)?

It appears that they are attacking via IP at this point as we cannot see a particular vhost or service that is under attack.

 

If this question asks for too much info, please feel free to delete or alter.

We'll not ever delete or alter your posts, it's better to answer your questions than to censor them.

 

I assume the FBI is involved with the problem as well?

I have contacted the local FBI Field Office however with it being Sunday there isn't much that can be done until the morning.

[Michael D.]

We've had so many people working on this I'm not sure if the attack has subsided or if our efforts to mitigate it have worked - but everything is online and operational.

[Michael D.]

If you have any specific questions concerning the attack you are welcome to post them here although I cannot guarantee that I will be able to respond quickly as I am generally the only one on the forums (due to there not being much activity) however if you put in a support ticket with your question you will get an answer very quickly as we always do our best to respond to support tickets quickly.

 

I would like to thank everybody for their patience and understanding in this matter. An e-mail has been dispatched to everybody on the Atlantis server - if you did not get the email here is a copy of it:

 

Subject: MDDHosting - DDoS Attack Fully Filtered

Hello,

 

At the time of me writing this message our network team has been able to fully filter out this attack which is an astonishing feat in and of itself. The attack peaked at around 15gbps and even with Cisco Guard around 250,000 requests per second were hitting the web server. We have done the best that we could do given the situation and I am proud to say that most hosts would be entirely down due to such an attack with no hopes of coming back online until the attack had actually subsided.

 

We are of course going to continue to watch the server very closely for any hit that the attacker is getting around the filters that we have put in place to block this attack. From what I was able to see in the logs the attacks were coming from 10,000+ compromised machines from all around the world. I am going to be going to the local FBI Field Office with log files and everything that I can provide to them without breaking our privacy policy or divulging any confidential client information.

 

I will say that we have learned quite a bit today from this attack and I will be personally implementing changes into some key systems and procedures that will not only help us prevent an attack such as this in the future but will also help us to more easily mitigate any such attack. DDoS attacks are a very difficult thing to be able to mitigate successfully and although the Atlantis server did experience substantial downtime today due to this attack I feel that over all we have done better than most other providers out there had they been placed in the same situation given the resources available to us.

 

By all means if you have any questions at all you are welcome to respond to this email and I will answer all of them as best as I can. I would like to personally thank you for your patience and understanding in this matter.

 

Thank you,

 

Michael Denney

[Lightpix]

Mike,

 

There are three academic centers across the country that are linked with the Justice Department in investigating cyber crimes such as yours. It seems twitter is being used for some of these attacks, such as with the Iranian government web sites a couple weeks ago.

 

You may be interested in contacting a nationally-known expert on DDoS attacks, Gary Warner, at gar@uab.edu as I believe he would want to know what you have learned and he may have some additional information for you. This attack was of sufficient severity and duration to be of concern to all who use the Internet.

 

Charles

[Michael D.]

Mike,

 

There are three academic centers across the country that are linked with the Justice Department in investigating cyber crimes such as yours. It seems twitter is being used for some of these attacks, such as with the Iranian government web sites a couple weeks ago.

 

You may be interested in contacting a nationally-known expert on DDoS attacks, Gary Warner, at gar@uab.edu as I believe he would want to know what you have learned and he may have some additional information for you. This attack was of sufficient severity and duration to be of concern to all who use the Internet.

 

Charles

They stepped the attack up to well beyond 20gbps today with well over 25,000 unique IP addresses bombarding our network. They did get an email through to me (the attacker) telling me to remove a site from our network and that the attack would subside.

 

SoftLayer had just contacted us to let us know that they were preparing to null-route our servers due to the severity of the attack and how much was even getting past Cisco Guard so we had little option but to remove the site and within moments of doing so the attacks stopped.

 

We have contacted this client directly to let them know the details of what was going on as well as refunded them in full - I really hate to do it but when it comes down to one site or our entire network I simply cannot afford to have our entire network down for days if not weeks - it would be the end of MDDHosting.

 

We are taking steps that will allow us to more effectively defend against attacks in the future as we have learned a lot during these attacks however with an attack as large as this last one - no amount of filtering or protection is going to work. If we were not at SoftLayer the entire Data Center's network would have certainly crashed at just about any other DC and there would have been no hope of ever coming back online until the issue was resolved.

 

If anybody has any questions let me know.

[Michael D.]

As it turns out there were actually two distinct DDoS attacks hitting the Atlantis server so I think we did pretty job keeping things as accessible as they were although that wasn't much.

 

We've certainly learned a lot during these incidents and are putting new policies and procedures in place to handle situations such as this to hopefully avoid as much downtime as was experienced.

 

I thank you all again for your patience during these incidents.

 

I believe the same attacker that was hitting our individual servers is hitting the SoftLayer core routers. I am not able to access the back-end management portal at SoftLayer at this time.

 

Even their PPTP and SSL VPN to the private network is down which is usually my backup to getting into their network to do things that need done: http://www.screen-shot.net/ss/3305711547840901119.png

 

I do not believe this will affect us directly at this point in time however I felt it pertinent to let everybody know.

[MjrNuT]

I wanted to say that I appreciate the efforts of MDD on this unfortunate event. For me, my site was early stage setup so impact was negligible, so MDD didn't have to worry about little 'ole me. I thank the staff for the informative updates of the whole situation. I don't really have anything to add on that front as I have no experience.

 

I do have a question though and sorry if it was mentioned already, but how/why MDD be a target for this?

[Michael D.]

I do have a question though and sorry if it was mentioned already, but how/why MDD be a target for this?

"We" weren't the target, just one of our clients and being that they were on one of our servers that means the attack hit our server almost as an innocent bystander because the attack wasn't directly focused on us or our server but instead the domain hosted on our server.