Jump to content


Photo

Network Outage 1/17/2019 - 5:56 AM to 6:54 AM


  • Please log in to reply
1 reply to this topic

#1 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 219 posts
  • Gender:Male

Posted 17 January 2019 - 07:59 AM

We received notifications at 5:56 AM that everything had gone offline.  We reached out to our facility and they were already aware and working to resolve the issue.

 

For clarity this was an issue upstream from us - our network and servers were online and operational but lost connectivity to the internet when the facility as a whole experienced a disruption.

 

The network came back online after 58 minutes and we are awaiting a Root Cause Analysis [RCA] from our facility regarding the network outage.  As soon as we have the cause of the outage we will make it available here.

 

This thread will be updated when we know more.


  • 1

#2 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,882 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 17 January 2019 - 09:08 AM

I don't have a formal RCA from the facility yet but I do expect one sometime today or tomorrow.

 

The short version is essentially that there was a mis-ordered routing policy that resulted in unexpected withdrawal of the default networking route when a single carrier was removed from the transit mix.

 

It is normal from time to time for the mix to change - say a specific carrier/transit provider is experiencing packet loss or network issues - the network will be removed from the mix to ensure performance.  In this case Hurricane Electric was being dropped from the transit due to issues with their network.

 

As this happened at the facility level it occurred on networking equipment we do not maintain nor have access to - unfortunately out of our control.  We are confident that this issue will not recur.

 

If you have any questions about any of this please feel free to reply here or to open a support ticket.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users