ericr Posted January 17, 2019 Report Share Posted January 17, 2019 We received notifications at 5:56 AM that everything had gone offline. We reached out to our facility and they were already aware and working to resolve the issue. For clarity this was an issue upstream from us - our network and servers were online and operational but lost connectivity to the internet when the facility as a whole experienced a disruption. The network came back online after 58 minutes and we are awaiting a Root Cause Analysis [RCA] from our facility regarding the network outage. As soon as we have the cause of the outage we will make it available here. This thread will be updated when we know more. 1 Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 17, 2019 Report Share Posted January 17, 2019 I don't have a formal RCA from the facility yet but I do expect one sometime today or tomorrow. The short version is essentially that there was a mis-ordered routing policy that resulted in unexpected withdrawal of the default networking route when a single carrier was removed from the transit mix. It is normal from time to time for the mix to change - say a specific carrier/transit provider is experiencing packet loss or network issues - the network will be removed from the mix to ensure performance. In this case Hurricane Electric was being dropped from the transit due to issues with their network. As this happened at the facility level it occurred on networking equipment we do not maintain nor have access to - unfortunately out of our control. We are confident that this issue will not recur. If you have any questions about any of this please feel free to reply here or to open a support ticket. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.