Jump to content


Photo

Facility-Wide Disruption of Connectivity

Resolved

  • Please log in to reply
14 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 28 February 2014 - 09:35 AM

Our data center facility, Handy Networks, is experiencing an extremely major DDoS attack that I believe is focused on their infrastructure.  Due to the nature of the attack and the size of the attack we unfortunately do not have an ETA nor is resolution of this issue within our direct control.  All staff at the facility are on-hand and actively working to resolve the issue as it's affecting everybody in the facility and not just us.

Do understand that *we* nor none of our customers are under attack but this attack is large enough to affect the entire facility.  Normally such an issue would be handled in 5 minutes or less but the duration of this incident further indicates the severity of this attack.

Connectivity is not completely lost but is intermittent as not all connections to the facility are affected - for example - I'm able to access the network fine in order to send you this message but my own connectivity is hit or miss.

Once this issue is resolved I'm sure the facility will issue an official RFO [Reason For Outage] which we will make available.

We're sorry for the trouble this issue may be causing you and we're sure our facility will get it sorted as quickly as humanly possible.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#2 gplurn

gplurn

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 28 February 2014 - 09:39 AM

Thanks for the info Mike
  • 0

#3 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 28 February 2014 - 09:45 AM

It looks like the first-line tech I spoke with initially was incorrect - it's not a DDoS issue but is indeed a networking issue.  Only a few customers, us included, are affected at this point - and not all connectivity is lost [we're pushing 90 MBPS currently while normal at this time is ~120 MBPS].  I've gotten with the Chief Technical Officer of the facility and they're addressing this issue.

 

I'll post more details once I have them but the focus is on restoring full connectivity at this time.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#4 mrintech

mrintech

    Newbie

  • Members
  • Pip
  • 12 posts
  • Gender:Male
  • Location:Bhopal, India

Posted 28 February 2014 - 10:27 AM

TFS, hope the issue gets sorted soon :)


  • 0

#5 Oro Jackson

Oro Jackson

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 28 February 2014 - 10:31 AM

My members (half of them) reported unable to reach the website, I knew something was up and checked mail (am subscribed to this thread!)

 

Hope the issue sorts out soon, good luck!


  • 0

#6 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 28 February 2014 - 10:32 AM

MDDHosting.com is hosted within the facility - the sheer fact that you can view and respond to this thread further goes to show not everybody is affected.

 

I have come to find out that the issue is a high-level Juniper issue [Handy Networks uses Juniper Routing and Switching] and their emergency support has been activated.  It will be resolved as quickly as humanly possible but I have no ETA.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#7 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 28 February 2014 - 11:18 AM

Unfortunately I have nothing new to provide at this point as those working on the issue are focused on fixing it and not focused on updating me [for obvious reasons].  We're not the only customer at the facility affected - I will do my best to update this thread if anything changes or I have more information to provide.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#8 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 28 February 2014 - 11:34 AM

THANK YOU EVERYBODY THAT SENT IN A TRACEROUTE.

 

We were able to identify the issue down to the Hurricane Electric and Telia transit providers.  We turned off those providers and all traffic is now routing over alternate unaffected routes - i.e. everybody should be back online at this point.

 

Understand due to the nature of this issue it is possible there will be further downtime/connectivity issues as there is no way for me to predict what changes have to be made to fully resolve the issue.  Turning off these transit providers is a band-aid over the symptoms and not a solution to the problem.  The goal was to bring everybody back online as quickly as possible.

 

We couldn't have isolated this issue without the traceroutes - thank you.

 

That said - if you are still experiencing connectivity issues do please email maintenance@ with a traceroute.


  • 3
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#9 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 28 February 2014 - 03:27 PM

MDDHosting.com is hosted within the facility...

 

Where is MDDHosting hosted?  Or rather.  Who hosts MDDHosting?


  • 0

#10 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 28 February 2014 - 04:11 PM

Nevermind, just re-read your first post which mentions it...


  • 0

#11 mrintech

mrintech

    Newbie

  • Members
  • Pip
  • 12 posts
  • Gender:Male
  • Location:Bhopal, India

Posted 01 March 2014 - 01:23 AM

Thanks for the update  :wub:


  • 0

#12 Oro Jackson

Oro Jackson

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 01 March 2014 - 01:53 AM

Is there any update on the status now? Or is it the same?


  • 0

#13 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 01 March 2014 - 02:11 AM

Sorry - it's been a long and arduous day and I forgot to post an update.  The issue was resolved when we dropped HE and Telia for us and our customers.  The issue was then resolved by the facility before HE and Telia were turned back on.
 
I will be posting the official Reason For Outage [RFO] once the facility has it available likely on Tuesday or Wednesday.
  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#14 Leah2

Leah2

    Member

  • Members
  • PipPip
  • 29 posts
  • Gender:Not Telling

Posted 01 March 2014 - 07:32 PM

Thanks for the update, since the change I have not recieved a pingdom downtime / uptime report. So all is well over here.


  • 0

Electronic Logic Concepts

 

“What is Your Digital Strategy? Websites Built With SEO First Practices”

 

www.ELC-SEO.com


#15 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 06 March 2014 - 07:25 AM

The official Reason for Outage has been provided to us by our upstream provider.  I'm attaching it to this thread.

Attached Files


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users