Jump to content


Photo

February Networking Outages - 02/12/16, 02/24/16 and 02/26/16

RFO Network Outage HandyNetworks Handy Networks

  • Please log in to reply
24 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 February 2016 - 03:42 PM

First and foremost I want to apologize for the unreliability of our network as of late.  While we do own and operate all of our servers/switches we do not run the entire network at the facility and do rely upon them to provide us the connectivity we pay them for.

 

We have been with Handy Networks in Denver, Colorado for many years now.  Until recently the network has always performed very well.  There have been instances where a huge DDoS attack would take us offline and in a few cases they were large enough to affect the whole facility but by and large things over the years were stable.

 

As of lately things have been less stable and while this has affected you it has affected us as well.

 

On 02/12/16 there was a fiber cut affecting the transport provider, Level3, between our two locations.  Unfortunately our new equipment was on the side of the transport that does not have direct internet access [it passes over the transport before leaving to the internet].  While this transport is supposed to be physically diverse across two routes - it is clear that something is wrong when a single fiber cut takes both routes down.  Handy Networks is still working with Level3 on this matter to determine why a single fiber cut took down both allegedly diverse routes but I am not sure if we'll ever have a real answer from L3 on that.

 

Handy Networks is in the process of installing a secondary transport provider with a diverse physical link and network but such things take time - at this point I'm hoping it will be available within 4 weeks.  Understand this isn't anything within our control and is entirely up to our facility to handle.

 

On 02/24/16 we lost all network connectivity including connectivity to HandyNetworks.com itself from 1:04 PM to about 1:17 PM.  We experienced another outage from 3:08 PM to about 3:16 PM.  At this point I do not have an official RFO or any details beyond that it was a networking issue at our facility outside of our control.  As soon as I have further details I will make them available.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#2 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 February 2016 - 03:45 PM

There will be scheduled networking interruptions this evening to address this - more information directly from our upstream provider, Handy Networks [Times are Mountain Time]
 

Emergency Network Maintenance Windows: Feb 24, 2016 @ 9:00PM - 1:00AM
We will be conducting emergency network maintenance this evening from February 24 @ 9:00PM - 1:00AM to address the underlying condition that has caused the two periods of packet loss and latency that were experienced earlier today.  During this time, you can expect to have several other periods of packet loss and latency.


Unfortunately this is entirely outside of our control. I will do my best to get the details of what needs to be changed and why as well as what the actual cause of the issues is but I do not have that information to provide at this time.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#3 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 154 posts
  • Gender:Not Telling

Posted 24 February 2016 - 04:12 PM

Thank you for keeping us informed and your honesty about what is going on.


  • 0

#4 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 February 2016 - 04:19 PM

Absolutely - I am sorry that I even had to write this post and that our customers have experienced issues.  I'm doing everything within my power to ensure things remain stable moving forward after this maintenance window but ultimately we do rely on our providers just like our customers rely on us.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#5 mcfrye

mcfrye

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 24 February 2016 - 04:28 PM

What time zone are the times for the emergency maintenance?


  • 0

#6 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,893 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 February 2016 - 04:33 PM

Mountain Time - it will be 11 PM to 3 AM ET.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#7 Rhody401

Rhody401

    Newbie

  • Members
  • Pip
  • 12 posts
  • Gender:Male
  • Location:Providence, RI USA
  • Interests:Consultant, Contractor, Beta Tester, IT Director.

Posted 24 February 2016 - 07:42 PM

Thanks for sharing the info.  This is a minor inconvenience at a good time of day, and it's good that they are addressing the issue instead of ignoring it.

 

I have only been a customer for 5 days, but so far I am very impressed with the stellar customer service.


  • 0

#8 AMC4x4

AMC4x4

    Newbie

  • Members
  • Pip
  • 24 posts

Posted 25 February 2016 - 01:11 AM

It's this kind of transparency and accountability that keeps me here, guys. Keep doing what you're doing. I know I'm not on one of your more expensive plans, but you've always treated me as a valued customer, and I really appreciate it. Just wanted to say thanks.


  • 0

#9 Vilandra

Vilandra

    Newbie

  • Members
  • Pip
  • 23 posts
  • Gender:Female
  • Location:Pittsburgh, PA
  • Interests:Chelsea FC!

Posted 25 February 2016 - 06:59 AM

I can't tell you how much I appreciate you keeping us informed like this. Thank you for all  you do! :)


  • 0

#10 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 221 posts
  • Gender:Male

Posted 26 February 2016 - 06:06 AM

I am adding the outage for February 2/26/2016 to this thread.

At this time our datacenter is investigating switch issues at the new location.  When I have further updates I will update this thread.


  • 0

#11 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 221 posts
  • Gender:Male

Posted 26 February 2016 - 06:15 AM

I am also looking into the secondary faults that are occurring on the servers where they are pingable but unable to display web pages.


  • 0

#12 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 221 posts
  • Gender:Male

Posted 26 February 2016 - 06:28 AM

I have located the cause of the current issues.  The network failure in the datacenter has included our connection to the SAN that we are using for our high speed storage.  I am awaiting updates from the datacenter.


  • 0

#13 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 221 posts
  • Gender:Male

Posted 26 February 2016 - 06:45 AM

They are still working on the core switch at the location. I will update as soon as I can.


  • 0

#14 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 221 posts
  • Gender:Male

Posted 26 February 2016 - 07:36 AM

We have isolated the fault and are working to reolve the issue with the SAN>  I am not able to provide a ETA as this is not a scheduled or planned fault. When I can provide a ETA I will gladly provide one on this thread.


  • 0

#15 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 221 posts
  • Gender:Male

Posted 26 February 2016 - 07:45 AM

I want to put out that tentatively all servers are up.  I am standing by for a cause of the failure of the SAN links.  


  • 0

#16 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 154 posts
  • Gender:Not Telling

Posted 26 February 2016 - 07:46 AM

I'm back online.


  • 0

#17 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 154 posts
  • Gender:Not Telling

Posted 26 February 2016 - 07:48 AM

I'm back online.


  • 0

#18 Tindell

Tindell

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 26 February 2016 - 07:52 AM

I'm also back online. Thank you for the continued updates.


  • 0

#19 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 221 posts
  • Gender:Male

Posted 26 February 2016 - 07:54 AM

We may need to reboot some or all of the servers to repair the underlying filesystem due to damage caused by the outage.  We will update this thread prior to doing so.


  • 0

#20 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 221 posts
  • Gender:Male

Posted 26 February 2016 - 08:08 AM

S3 and S4 need emergency work to repair the filesystems so they can function.  I am working on S3 right now.


  • 0





Also tagged with one or more of these keywords: RFO, Network Outage, HandyNetworks, Handy Networks

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users