[Resolved] Fresco Outage - DDoS

[Michael D.]

The Fresco server is currently seeing 95 to 98% packet loss due to a DDoS attack and our networking team is working on this. We expect to have it mitigated within 5 to 10 minutes however nothing is set-in-stone as this is preliminary.

 

If you have any questions, let us know.

[Michael D.]

We've un-routed the targeted IP from the server temporarily while the networking team works on the issue to restore service to everybody else.

[Juan]

Thanks!

[Michael D.]

The attack has been mitigated (as of a few minutes ago). Here are graphs for those who may be curious. These are our network-wide graphs and are not specific to the server but it's obvious when the attack starts and when it was resolved.

 

 

http://www.screen-shot.net/2011-05-14_1233_traf.png

http://www.screen-shot.net/2011-05-14_1233_packets.png

[Juan]

Very fast response time by MDD (10 mins maybe less?) Thank you, Mike!

[Michael D.]

We're now seeing another attack to the same server, however on a completely different/unrelated IP address. We're working with our networking team to mitigate this attack.

[Vilandra]

Mike I'm getting this error now - assuming it's part of this, but just in case it means something:

 

The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression.

[Michael D.]

Mike I'm getting this error now - assuming it's part of this, but just in case it means something:

That generally means that you're forcing GZIP compression, when the server already gzip compresses content on-demand. Do open a ticket as this would be unrelated.

 

On the subject of the attack - the IP that came under attack was null routed (i.e. routed to nowhere/black hole) and we're shifting accounts off of that IP as quickly as possible in small blocks to fresh IPs. This will not only be bringing sites affected back online but will also allow us to identify the target of the attack (should the attack shift with the account, which it usually does).

 

If you have any questions about the attack, let us know.

[Michael D.]

All accounts that were on the affected IP have been shifted off of the IP so service should be fully restored to everybody. It is likely that the attack will shift to the new IP of whichever account was originally targeted at which point we will further isolate the target to fully mitigate the attack.

[Vilandra]

Thank you for your hard work, Mike

[Michael D.]

Thank you for your hard work, Mike

Absolutely, myself and our networking team I wish the internet wasn't such a hostile place.

 

Not that any of you have compromised computers (surely we all have secured systems!) but if you're not actively doing this, make sure you do it:

• Make sure you have quality up-to-date virus scanning running.
  
• Make sure that you do periodic scans for malware such as with malwarebytes.
  
• Make sure that you keep your operating system up to date.
  
• If you're not running a server or something that requires your computer to be on, do turn it off when you're not using it.
  

 

DDoS attacks come from compromised systems that are left online and do not have the necessary security patches, virus scanning, etc... If everybody followed these simple guidelines a vast majority of DDoS attacks wouldn't be possible as the computers that were a part of the botnets would be secured.