Jump to content
MDDHosting Forums

Network Outage 1/17/2019 - 5:56 AM to 6:54 AM

Recommended Posts

We received notifications at 5:56 AM that everything had gone offline. We reached out to our facility and they were already aware and working to resolve the issue.


For clarity this was an issue upstream from us - our network and servers were online and operational but lost connectivity to the internet when the facility as a whole experienced a disruption.

The network came back online after 58 minutes and we are awaiting a Root Cause Analysis [RCA] from our facility regarding the network outage. As soon as we have the cause of the outage we will make it available here.


This thread will be updated when we know more.

  • Upvote 1
Link to comment
Share on other sites

I don't have a formal RCA from the facility yet but I do expect one sometime today or tomorrow.


The short version is essentially that there was a mis-ordered routing policy that resulted in unexpected withdrawal of the default networking route when a single carrier was removed from the transit mix.


It is normal from time to time for the mix to change - say a specific carrier/transit provider is experiencing packet loss or network issues - the network will be removed from the mix to ensure performance. In this case Hurricane Electric was being dropped from the transit due to issues with their network.


As this happened at the facility level it occurred on networking equipment we do not maintain nor have access to - unfortunately out of our control. We are confident that this issue will not recur.


If you have any questions about any of this please feel free to reply here or to open a support ticket.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...