Michael D. Posted February 28, 2014 Report Share Posted February 28, 2014 Our data center facility, Handy Networks, is experiencing an extremely major DDoS attack that I believe is focused on their infrastructure. Due to the nature of the attack and the size of the attack we unfortunately do not have an ETA nor is resolution of this issue within our direct control. All staff at the facility are on-hand and actively working to resolve the issue as it's affecting everybody in the facility and not just us.Do understand that *we* nor none of our customers are under attack but this attack is large enough to affect the entire facility. Normally such an issue would be handled in 5 minutes or less but the duration of this incident further indicates the severity of this attack.Connectivity is not completely lost but is intermittent as not all connections to the facility are affected - for example - I'm able to access the network fine in order to send you this message but my own connectivity is hit or miss.Once this issue is resolved I'm sure the facility will issue an official RFO [Reason For Outage] which we will make available.We're sorry for the trouble this issue may be causing you and we're sure our facility will get it sorted as quickly as humanly possible. 1 Quote Link to comment Share on other sites More sharing options...
gplurn Posted February 28, 2014 Report Share Posted February 28, 2014 Thanks for the info Mike Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 28, 2014 Author Report Share Posted February 28, 2014 It looks like the first-line tech I spoke with initially was incorrect - it's not a DDoS issue but is indeed a networking issue. Only a few customers, us included, are affected at this point - and not all connectivity is lost [we're pushing 90 MBPS currently while normal at this time is ~120 MBPS]. I've gotten with the Chief Technical Officer of the facility and they're addressing this issue. I'll post more details once I have them but the focus is on restoring full connectivity at this time. Quote Link to comment Share on other sites More sharing options...
mrintech Posted February 28, 2014 Report Share Posted February 28, 2014 TFS, hope the issue gets sorted soon Quote Link to comment Share on other sites More sharing options...
Oro Jackson Posted February 28, 2014 Report Share Posted February 28, 2014 My members (half of them) reported unable to reach the website, I knew something was up and checked mail (am subscribed to this thread!) Hope the issue sorts out soon, good luck! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 28, 2014 Author Report Share Posted February 28, 2014 MDDHosting.com is hosted within the facility - the sheer fact that you can view and respond to this thread further goes to show not everybody is affected. I have come to find out that the issue is a high-level Juniper issue [Handy Networks uses Juniper Routing and Switching] and their emergency support has been activated. It will be resolved as quickly as humanly possible but I have no ETA. 1 Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 28, 2014 Author Report Share Posted February 28, 2014 Unfortunately I have nothing new to provide at this point as those working on the issue are focused on fixing it and not focused on updating me [for obvious reasons]. We're not the only customer at the facility affected - I will do my best to update this thread if anything changes or I have more information to provide. 1 Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 28, 2014 Author Report Share Posted February 28, 2014 THANK YOU EVERYBODY THAT SENT IN A TRACEROUTE. We were able to identify the issue down to the Hurricane Electric and Telia transit providers. We turned off those providers and all traffic is now routing over alternate unaffected routes - i.e. everybody should be back online at this point. Understand due to the nature of this issue it is possible there will be further downtime/connectivity issues as there is no way for me to predict what changes have to be made to fully resolve the issue. Turning off these transit providers is a band-aid over the symptoms and not a solution to the problem. The goal was to bring everybody back online as quickly as possible. We couldn't have isolated this issue without the traceroutes - thank you. That said - if you are still experiencing connectivity issues do please email maintenance@ with a traceroute. 3 Quote Link to comment Share on other sites More sharing options...
Brad Posted February 28, 2014 Report Share Posted February 28, 2014 MDDHosting.com is hosted within the facility... Where is MDDHosting hosted? Or rather. Who hosts MDDHosting? Quote Link to comment Share on other sites More sharing options...
Brad Posted February 28, 2014 Report Share Posted February 28, 2014 Nevermind, just re-read your first post which mentions it... Quote Link to comment Share on other sites More sharing options...
mrintech Posted March 1, 2014 Report Share Posted March 1, 2014 Thanks for the update Quote Link to comment Share on other sites More sharing options...
Oro Jackson Posted March 1, 2014 Report Share Posted March 1, 2014 Is there any update on the status now? Or is it the same? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted March 1, 2014 Author Report Share Posted March 1, 2014 Sorry - it's been a long and arduous day and I forgot to post an update. The issue was resolved when we dropped HE and Telia for us and our customers. The issue was then resolved by the facility before HE and Telia were turned back on. I will be posting the official Reason For Outage [RFO] once the facility has it available likely on Tuesday or Wednesday. 1 Quote Link to comment Share on other sites More sharing options...
Leah2 Posted March 2, 2014 Report Share Posted March 2, 2014 Thanks for the update, since the change I have not recieved a pingdom downtime / uptime report. So all is well over here. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted March 6, 2014 Author Report Share Posted March 6, 2014 The official Reason for Outage has been provided to us by our upstream provider. I'm attaching it to this thread.2014-Feb-28-Network-RFO.pdf Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.