Jump to content
MDDHosting Forums

February Networking Outages - 02/12/16, 02/24/16 and 02/26/16


Recommended Posts

First and foremost I want to apologize for the unreliability of our network as of late. While we do own and operate all of our servers/switches we do not run the entire network at the facility and do rely upon them to provide us the connectivity we pay them for.

 

We have been with Handy Networks in Denver, Colorado for many years now. Until recently the network has always performed very well. There have been instances where a huge DDoS attack would take us offline and in a few cases they were large enough to affect the whole facility but by and large things over the years were stable.

 

As of lately things have been less stable and while this has affected you it has affected us as well.

 

On 02/12/16 there was a fiber cut affecting the transport provider, Level3, between our two locations. Unfortunately our new equipment was on the side of the transport that does not have direct internet access [it passes over the transport before leaving to the internet]. While this transport is supposed to be physically diverse across two routes - it is clear that something is wrong when a single fiber cut takes both routes down. Handy Networks is still working with Level3 on this matter to determine why a single fiber cut took down both allegedly diverse routes but I am not sure if we'll ever have a real answer from L3 on that.

 

Handy Networks is in the process of installing a secondary transport provider with a diverse physical link and network but such things take time - at this point I'm hoping it will be available within 4 weeks. Understand this isn't anything within our control and is entirely up to our facility to handle.

 

On 02/24/16 we lost all network connectivity including connectivity to HandyNetworks.com itself from 1:04 PM to about 1:17 PM. We experienced another outage from 3:08 PM to about 3:16 PM. At this point I do not have an official RFO or any details beyond that it was a networking issue at our facility outside of our control. As soon as I have further details I will make them available.

Link to comment
Share on other sites

There will be scheduled networking interruptions this evening to address this - more information directly from our upstream provider, Handy Networks [Times are Mountain Time]

Emergency Network Maintenance Windows: Feb 24, 2016 @ 9:00PM - 1:00AM
We will be conducting emergency network maintenance this evening from February 24 @ 9:00PM - 1:00AM to address the underlying condition that has caused the two periods of packet loss and latency that were experienced earlier today. During this time, you can expect to have several other periods of packet loss and latency.


Unfortunately this is entirely outside of our control. I will do my best to get the details of what needs to be changed and why as well as what the actual cause of the issues is but I do not have that information to provide at this time.

Link to comment
Share on other sites

Absolutely - I am sorry that I even had to write this post and that our customers have experienced issues. I'm doing everything within my power to ensure things remain stable moving forward after this maintenance window but ultimately we do rely on our providers just like our customers rely on us.

Link to comment
Share on other sites

Thanks for sharing the info. This is a minor inconvenience at a good time of day, and it's good that they are addressing the issue instead of ignoring it.

 

I have only been a customer for 5 days, but so far I am very impressed with the stellar customer service.

Link to comment
Share on other sites

I would also appreciate info on S1 today's outage reasons for 2 hours.

The network disruption to our storage platform is the cause for the outage on S1 I believe.

 

We're currently working with our storage vendor as well as our facility to determine exactly how this happened so we can prevent it. We are using a fully dual redundant network where any part of it can fail and everything remains online. This failure was outside of our network border, however, it still caused connectivity issues and that's why we're still investigating.

 

Once I know why it happened I will post it here. I do not, as of yet, have that diagnosis.

Link to comment
Share on other sites

 Share

×
×
  • Create New...