Jump to content


MikeDVB

Member Since 27 Sep 2008
Offline Last Active Jan 24 2019 11:38 AM

Posts I've Made

In Topic: Network Outage 1/17/2019 - 5:56 AM to 6:54 AM

17 January 2019 - 09:08 AM

I don't have a formal RCA from the facility yet but I do expect one sometime today or tomorrow.

 

The short version is essentially that there was a mis-ordered routing policy that resulted in unexpected withdrawal of the default networking route when a single carrier was removed from the transit mix.

 

It is normal from time to time for the mix to change - say a specific carrier/transit provider is experiencing packet loss or network issues - the network will be removed from the mix to ensure performance.  In this case Hurricane Electric was being dropped from the transit due to issues with their network.

 

As this happened at the facility level it occurred on networking equipment we do not maintain nor have access to - unfortunately out of our control.  We are confident that this issue will not recur.

 

If you have any questions about any of this please feel free to reply here or to open a support ticket.


In Topic: Processor Upgrades - Faster Single-Threaded Performance & Higher Overall...

11 January 2019 - 08:28 PM

All processors in all compute nodes are fully upgraded :).  All shared, reseller, and VPS nodes will see the benefits of this upgrade.


In Topic: November 18, 2018 - DDoS Attack

20 November 2018 - 09:40 PM

Any update on this?  Thanks.

Nothing to update. The attacks were absorbed and ended. 


In Topic: November 18, 2018 - DDoS Attack

18 November 2018 - 02:19 PM

First thing I did was go to the twitter channel.  I did not see anything so I opened a ticket.

 

I think it is better to announce it there. 

It's only affecting a very extremely tiny subset of clients - so I didn't want to get everybody concerned about it unless the attack gets to the point that we can't manage it or have to take drastic steps.

 

To be completely straightforward - these attacks aren't big enough to take anything we have offline - but our facility is proactively trying to protect us.  I've asked that they stop doing this [stop null-routing IPs] which will result in things staying online.

 

If the attack does get to the point that our infrastructure can't handle it - we do have other options available to us other than de-routing IPs and we'll use those first.

 

It's been a LONG time since we've had any DDoS attacks much less of this magnitude and our infrastructure has changed substantially since the last time so our facility is doing their best to protect us and is being a bit ... too overprotective :).


In Topic: November 18, 2018 - DDoS Attack

18 November 2018 - 02:03 PM

For clarity there is a lot of detail I am not providing here in this thread due to the nature of the attacks - but we are aware of them and working to resolve them as they change and adapt.