Unplanned outage 2/21/2017

[ericr]

We do not have full details at this time. However, the initial report is that while offloading our BGP announcements from the new 'core' to handy networks borders, there was an issue that prevented certain prefixes from being announced to handy networks upstream providers and peers.

[Michael D.]

Our facility scheduled network maintenance for Monday, Februrary 20 and Tuesday, Februrary 21 to replace and upgrade core network infrastructure.

 

On Monday night the maintenance went exactly as planned and no outages occurred. Unfortunately last night on the second and final day of the maintenance window everything didn't go 100% as-planned resulting in an unplanned and unexpected outage.

 

During the network maintenance last night there was a technical issue that prevented certain prefixes [iP ranges] from being announced upstream to the facility's providers and peers.

 

What this means is that while our servers and our network and servers were powered up and functional - some of the IP addresses we use were not properly announced to the internet. The IP ranges that were not properly announced essentially became un-routeable on the internet resulting in a loss of connectivity on those IP ranges.

 

Not every server or IP we have was affected but a large portion of our IP ranges were. I don't have any additional details at this time as to what exactly caused the outage. Our facility will be providing us with additional details as to why the outage occurred soon and once I know the specifics I will post them here.

 

We apologize for any trouble this caused you and we look forward to a stable network on this new and upgraded hardware infrastructure.