Jump to content


Photo

Connectivity issues over Telia and Hurricane Electric [Transit Providers]

Resolved

  • Please log in to reply
5 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 09 June 2016 - 01:40 PM

We were alerted within about a minute of connectivity issues between our network and Europe.
 
Upon investigation we found that a transit provider, Telia, is having issues.
 
We've removed Telia from our bandwidth mix and have reached out to them concerning the matter.  It will take a few minutes for things to stabilize but if you continue having issues do please open a ticket and provide a traceroute to your site as well as your IP address [ https://www.mddhosti.../whatismyip.php/ https://www.google.c...q=what is my ip ].
 
We apologize for the trouble this has caused you and hopefully we'll get a cause analysis from Telia or at least some details as to why it happened.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#2 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 09 June 2016 - 02:13 PM

We determined that both Telia and Hurricane Electric are experiencing major issues.

 

While you may not be able to reach us/your site - the majority of the word can.

 

Our networking department is in contact with both transit providers.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#3 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 09 June 2016 - 02:29 PM

The long and short of it is that there was a major outage with a major internet backbone provider that caused connectivity issues for some customers. This failure was well outside of our border [in New York, apparently] and well outside of our direct control.

Our networking department did reach out to Telia and HE and you will see below quoted the information we received from our networking department concerning the matter.

We have experienced a loss of connectivity due to a major outage with our carrier Telia, which they have confirmed. The issue was amplified by the fact that another provider, HE, also uses Telia. On our request after they received confirmation from Telia, HE de-peered with Telia until the issue is resolved.

We received these responses from Telia:

"Dear Customer,

This is part of a larger outage that is currently being investigated.

Kind Regards,
Luis Nuñez
Customer Care, Data & Infrastructure"

Then:
"We are currently experiencing issue with a backbone router in New York."


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#4 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 09 June 2016 - 03:23 PM

We have recieved an update from Telia:

Dear Customer,

Our core team has resolved an issue on our backbone causing U.S. customers packet loss. The root cause analysis will follow and we expect no further packet loss due to this issue.


We do not expect there to be further issues and are now marking this as resolved.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#5 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 10 June 2016 - 01:38 PM

Telia released their official RFO [Reason For Outage] and here are those details:

Dear Customer,

This is a Reason for Outage Report with details regarding the case you have opened with TeliaSonera International Carrier.

Country: United States
TeliaSonera Case Reference: 00563796
Network Impact: Packet loss on the Telia Carrier U.S. backbone.
Case Opened: 6/9/2016 7:00 PM (After the issue had begun)
Case Ready for Service: 6/9/2016 7:23 PM

Reason for Outage: Incorrect ISIS metric and multiple commits while turning up new Telia Carrier backbone links in Dallas caused a loop of reconverging BGP and ISIS protocols. This put a very high CPU load on our U.S. routers and caused some trans-Atlantic congestion.
Actions Taken: The nyk-bb1 inner-core router (New York) was the first router to show real problems when we received alarms indicating packet loss on transit-Atlantic traffic together with high CPU utilization. The router was taken out of service. Further investigation revealed that the root cause was too many commits by Implementation while turning up new backbone links on dls-b22 (Dallas), along with an incorrect metric. The configuration in dls-b22 was rolled back to alleviate the problem and nyk-bb1 has been put back in service. This resolution is permanent and the will be no further loss related to this issue
Additional Information: Telia Implementation team is making significant changes to their way of working to mitigate this from happening in the future.


Please note that all the time stamps given above are in UTC unless otherwise stated.

Please bear in mind that this was a major issue with the internet itself and one of it's larger backbone providers. This was not within our power to detect, prevent, or resolve.

We do apologize for any trouble this outage caused you.
  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#6 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 June 2016 - 10:41 AM

CloudFlare has an in-detail blog post on the issues with Telia Carrier - the ones that affected us and our customers as well.

We found it very interesting so I'm linking to it here:
https://blog.cloudfl...nings-incident/

It seems it was human error at Telia that caused these issues...
http://www.theregist...ive_net_outage/
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users