Jump to content


Highest Reputation Content


#7229 Major Outage - 09/21/18 - 09/24/2018

Posted by MikeDVB on 24 September 2018 - 06:28 PM

We performed some changes on the storage in both backup servers.  The fast backup server is now doing 4,770 GB/hour data rates while it was doing 500gb/hour.  The old backup server has been doing 318 gb/hour.  We're watching to see how much of an improvement the old backup server gets as a result of these changes too.

 

The bottle neck is getting the data from the old backup server to the fast one - but this process is substantially faster than restoring directly from the old backup server which was only doing about 50gb/hour.


  • 5


#7227 Major Outage - 09/21/18 - 09/24/2018

Posted by MikeDVB on 24 September 2018 - 06:14 PM

We upgraded a storage component on the new Fast Backup server and we are seeing a huge increase in performance over what we were already seeing.  This should cut down the total restoration time - we will have updated ETAs once we have conducted some more transfers.


  • 5


#7219 Major Outage - 09/21/18+ - Client Discussion

Posted by MisterNeutron on 24 September 2018 - 05:22 PM

How will the damages incurred be compensated ?

 

I am running a webshop generating quite a bit of revenue and 4 - 5 days without any sales will cost me at least US$ 5000 (direct damages).

Read the terms of service: https://www.mddhosting.com/tos.php

 

I don't think there's a web host on the planet that will compensate people for lost income due to downtime. They couldn't - a single 6-hour outage would effectively put them out of business. Does the electric company compensate businesses for lost revenue when the power is out for a couple of days?

 

Last but not least, if your site is generating that much income, do you think using a bargain shared hosting service is a smart business decision? Or should you perhaps be paying more for hosting that has more redundancy, and quicker disaster recovery? Or should you perhaps maintain two hosting services, with the ability to change your nameservers in a heartbeat, and have all of your business directed to the secondary service in case of unfortunate problems like this one?


  • 5


#7372 Major Outage - 09/21/18+ - Client Discussion

Posted by SarisIsop on 26 September 2018 - 03:51 AM

All up and running on S5

 

Have lost a few posts on our forum, but it's no big deal and we are just pleased to be back on-line.

 

I would like to say a big thank you to Michael and all the staff and helpers at MDD for getting us out of this. No it shouldn't have happened, but it did and you all rolled up your sleeves on got on with.

 

Michael should be commended for transparency and honesty to his clients, it takes guts to take it on the chin but he has proven that he will always put his customers first.

 

I also want to shout out to the person that made this happen. Don't dwell on it, it's the past and you can not change that. Dust yourself off, hold your head up high and carry on your journey into the future.

 

We are all to blame and I for one will be taking regular cPanel backups, had I have done that I would have been back on-line within a couple of hours instead of waiting till today. Lesson learnt.

 

I will not be leaving MDD. I'm only a small fish but they have always looked after me, sometimes far beyond what I pay them for. I also will not be requesting any refund/compensation as I have lost no money over this and I feel you will be losing enough.

 

I know all of you at MDD have still a lot of work ahead of you and yes you will undoubtedly be on the big side of losing, but this episode has made you stronger and I still have full faith in you. 

 

Sorry to those of you that have been posting your anger especially Avatar for telling me I will just have to "Be Annoyed" I do understand as I once ran my own business, but I just wanted the original thread to be updates and not questions or complaints. Anyway Michael did eventually split the thread and lock it, something I would recommend doing in future.

 

Good luck to everyone whichever way you have decided to go, and thank you MDD.


  • 4


#7230 Major Outage - 09/21/18 - 09/24/2018

Posted by MikeDVB on 24 September 2018 - 07:06 PM

R1 is restoring now, should be done soon.

 

R4 got finished faster than expected and we should be restoring it soon.

 

S2 is also close to being ready to get restored.


  • 4


#7395 Major Outage - 09/21/18+ - Client Discussion

Posted by AMC4x4 on 26 September 2018 - 11:45 AM

Just a note in response to "newbie" ... I, too, want to see MDDHosting grow and prosper.  I've used their services many years and referred many people.  But a couple of things:

 

1.  This is 2018, not the mid-2000s when those generalized and generic terms were drafted. 

 

2.  This series of devastating technical errors is more akin to what happened a decade ago. This is 2018.

 

3.  This is not so much about tech stuff, this is about company reputation ... how a person operates their business in 2018.  Perception about a company's reputation is paramount, and Michael's lengthy emails, now reposted on industry websites, won't help MDDHosting's reputation.  When I read them, I thought, "oh, no! Just say the service is down and it will be fixed."

 

4.  Just this week, MDDHosting has disappeared from some rating services as a "top" site host.

 

5.  Why not use your name?

 

Why so hostile? My name is Aaron Clow and I've been an MDD customer since maybe 2013 or so? I have two accounts with them - a corporate account for the company I work for and another account for my personal projects. If you care to look, many of us are "Newbie's" because MDD has had so few issues over the years that we haven't had to post here. "Newbie" is a classification, not my name. Also, when we sign up for a forum account, it doesn't put your name in as your username. You pick one. So I picked one. 

 

If you prefer a hosting company that hides their activities behind a simple "service is down," and that's all the information you require when something untoward happens, I'm sure there are plenty of cheap places you can go for that kind of service. That's not how I do business. I'm forthright with people I speak to, and the reason I'm still with MDD is precisely because they do business this way. Look at how many people are still here and who say they aren't leaving. Why do you think that is?

 

So what's your name? Do people call you "Dehenderson?"

 

Cheers 


  • 3


#7384 Major Outage - 09/21/18+ - Client Discussion

Posted by Dehenderson on 26 September 2018 - 09:58 AM

Removed by Request


  • 3


#7297 Major Outage - 09/21/18+ - Client Discussion

Posted by npad69 on 25 September 2018 - 11:19 AM

Sadly - human error - and then a chain of unfortunate circumstances got us to where we ended up. A chain that we will be breaking in numerous places so that this cannot happen again.

Hi Mike, not to divert the topic but I'm both curious and concerned about the person responsible for this error? Honestly, the first thing that came into my mind when things got a bit serious was how this admin would have affected him/her psychologically. I don't know.. I would have had some kind of nervous breakdown or something if it was me. Hows he/she holding up BTW? I hope he/she is ok.


  • 3


#7293 Major Outage - 09/21/18+ - Client Discussion

Posted by Big Dan on 25 September 2018 - 10:52 AM

As I understand it we were all very lucky that you had a another (off-site) backup of the data. If it were only the snapshots every one would have been up the creek without a paddle. A lot of hosts would simply point to their 'suicide clause' of maintain and only rely on your own backups. You have that clause in your TOS but still went the extra mile. 

 

No one wants down time and it's never easy especially when you have clients hooting and hollering at you over something you really cannot control. It happens and it sucks but frankly this turned into a good learning opportunity. 

 

I learned that I had grown complacent with MDD's stellar uptime and general lack of headaches compared to other hosts. I wasn't pulling down backups as often as I should have. That's on me. 

 

Secondly, only some of my clients are on Cloud Flare. All my clients are going to use Cloud Flare in the future. With Cloud Flare I can at the very least point email at another host with minimal downtime until MDD restores. 

 

Current backups + Cloud Flare means I can restore service to another server pretty much as fast as I'm able to upload them.

 

For additional piece of mind I'm going to keep a 'hot' account on another server with email forwarders setup for all clients. This way I can just do an MX switch at Cloud Flare to restore email (the life blood of many client's businesses) ASAP then work on restoring the websites. 

 

Mike and the team still have my vote of confidence even more so seeing how they handled this outage. I really should've had this disaster plan in place already. 


  • 3


#7279 Major Outage - 09/21/18+ - Client Discussion

Posted by MikeDVB on 25 September 2018 - 09:49 AM

Managed to get my S5 server website back online yesterday by moving to another host a couple days ago.  Had I moved when my gut told me to, as soon as all this happened, I could have been back online already on Sunday.  Lesson learned...  When things this serious go wrong, grab another host for a month right away and take shelter.  Or in my case...  Move.

If you could have moved - then we could have re-created your account and you could have restored in-place.  Whatever you moved to another provider could have been quickly brought back online with us.

 

The server has been online since Friday - the only thing we've been working on is copying our backup data over.  Any clients that have their own backups have been online for days already.


  • 3


#7221 Major Outage - 09/21/18 - 09/24/2018

Posted by MikeDVB on 24 September 2018 - 05:51 PM

We are starting the restoration of the R1 server now.


  • 3


#7139 Major Outage - 09/21/18+ - Client Discussion

Posted by cziv on 24 September 2018 - 09:54 AM

GUYS DO NOT POST SPECIFIC PROBLEMS HERE.

 

OPEN A TICKET !!!!!


  • 3


#4992 Facility-Wide Disruption of Connectivity

Posted by MikeDVB on 28 February 2014 - 11:34 AM

THANK YOU EVERYBODY THAT SENT IN A TRACEROUTE.

 

We were able to identify the issue down to the Hurricane Electric and Telia transit providers.  We turned off those providers and all traffic is now routing over alternate unaffected routes - i.e. everybody should be back online at this point.

 

Understand due to the nature of this issue it is possible there will be further downtime/connectivity issues as there is no way for me to predict what changes have to be made to fully resolve the issue.  Turning off these transit providers is a band-aid over the symptoms and not a solution to the problem.  The goal was to bring everybody back online as quickly as possible.

 

We couldn't have isolated this issue without the traceroutes - thank you.

 

That said - if you are still experiencing connectivity issues do please email maintenance@ with a traceroute.


  • 3


#7393 Major Outage - 09/21/18+ - Client Discussion

Posted by AMC4x4 on 26 September 2018 - 11:07 AM

 

To the person who started a crowd funding initiative on behalf of MDDHosting, just let me express that MDDHosting is not the victim, all of the company's clients are.  All of the company's clients who trusted that MDDHosting would professionally manage their websites were injured in one way or the other.  If there is, indeed, a crowd funding initiative, then the recipients of any funds should be MDDHosting's customers, the people who suffered and were inconvenienced.  Here's why -- what happened was not an "act of God." It was not a server failure or a datacenter fire.  

 

What happened was human incompetence ... a inexcusable human mistake ... a self-inflicted error ... or, a "man made disaster" as a lawyer friend calls it.  It should not have happened.  And, liability (responsibility) is with MDDHosting. 

 

Perhaps even worse is the fact that MDDHosting’s backup system - which would have enabled expeditious recovery - had been deactivated …. turned OFF.  Another “man made disaster.”

 

You and I learned about this through at least two extraordinarily lengthy emails from Michael Denney, including details on the incorrect code entered that deleted everything.  As someone who respects Michael, I wish he had not spent time writing those emails and more time on finding a fix.  Additionally, those emails expose the company to untold legal exposure at this point.

 

As for our sites, we are discussing whether to throw in the towel.  Our primary site is BoomerCafe.com, a baby boomer story sharing initiative that was launched in 1999.  Nearly 20 years of equity has been wiped out in less than a week.  I hope Michael and MDDHosting will find a professional and responsible way to make us … ALL of us … whole with our respective damages due to this disaster that was clearly … CLEARLY … the result of human failure.

 

 

1) What happened was actually a series of events, not a single event. It might have seemed like a single event, but it wasn't *backups* that were offline, it was snapshots. The backup server was working, slow as it was, and thank god for that. Otherwise many of us would have been well and truly $crewed. 

2) I agree about the GoFundMe. While many of us are still fans of Michael and MDD, this has cost us as well as them, and while I'm keeping most of my sites with MDD, I'm not donating to the GoFundMe because we are all out something here. I spent money to move one of my sites elsewhere that I had an UpdraftPlus backup for. Regular backups and backup testing are a must for anyone who maintains content and deals with IT. I only had those for some of my sites. That's on me. If I had regular cPanel backups, I would have been back online within 24 hours at most. Again, shared responsibility. I believe the ToS say you are responsible for your own backups.

3) I believe the ToS covers Mike and MDD from "untold legal exposure" because at the end of the day, YOU are responsible for backing up your data.

4) Writing those letters did not take away from time spent on restoration. There was a well-documented bottleneck with the backup server that gave Mike and his crew time to answer tickets, keep us updated, etc. I'm sure if they could have sped that copying process along, they would have been happy to have spent time on restoration rather than taking the time to patiently respond to all our questions.

5) I see your site is up, with content as recent as the 18th. How do you figure "Nearly 20 years of equity has been wiped out in less than a week." Just curious


  • 2


#7392 Major Outage - 09/21/18+ - Client Discussion

Posted by digibread on 26 September 2018 - 11:00 AM

 

To the person who started a crowd funding initiative on behalf of MDDHosting, just let me express that MDDHosting is not the victim, all of the company's clients are.  All of the company's clients who trusted that MDDHosting would professionally manage their websites were injured in one way or the other.  If there is, indeed, a crowd funding initiative, then the recipients of any funds should be MDDHosting's customers, the people who suffered and were inconvenienced.  Here's why -- what happened was not an "act of God." It was not a server failure or a datacenter fire.  

 

What happened was human incompetence ... a inexcusable human mistake ... a self-inflicted error ... or, a "man made disaster" as a lawyer friend calls it.  It should not have happened.  And, liability (responsibility) is with MDDHosting. 

 

Perhaps even worse is the fact that MDDHosting’s backup system - which would have enabled expeditious recovery - had been deactivated …. turned OFF.  Another “man made disaster.”

 

You and I learned about this through at least two extraordinarily lengthy emails from Michael Denney, including details on the incorrect code entered that deleted everything.  As someone who respects Michael, I wish he had not spent time writing those emails and more time on finding a fix.  Additionally, those emails expose the company to untold legal exposure at this point.

 

As for our sites, we are discussing whether to throw in the towel.  Our primary site is BoomerCafe.com, a baby boomer story sharing initiative that was launched in 1999.  Nearly 20 years of equity has been wiped out in less than a week.  I hope Michael and MDDHosting will find a professional and responsible way to make us … ALL of us … whole with our respective damages due to this disaster that was clearly … CLEARLY … the result of human failure.

 

 

Sir, I started the fundraiser and I don't mean any disrespect to you, but you (and anyone else thinking like you) clearly did not read your end user agreement when you signed up. Your Data and Access to it, is Your Responsibility! That goes with any hosting provider out there. Your loss in equity is your fault and yours alone. There are so many ways (for site owners, hosting resellers, etc.) to mitigate what all of us experienced here that there is no excuse. The final responsibility will always and does now stop with you.

 

Human error will always play into anything we do as businesses and most companies could give a $h1t about their customers. MDD is not like that at all and Michael's transparency is refreshing and much appreciated. MDD is a small business with an exceptional owner with a young family and responsibilities to employees. They clearly have responsibilities to their customers and have performed to the best of their abilities given the situation. 

 

I had over 32 customers down for 4 days, including a National Food Chain and not one single loss or complaint like yours. You see, even if MDD was a terrible service provider and didn't have any backup I could have quickly setup new hosting and them all back online within a 24 and the most critical within a few. However, each one trusted me that MDD would come through and they did. 

 

You are correct in this: MDD is not a victim, but sir, neither are you! The fundraisers that piss me off are the ones for political douche bags. And they get untold amounts of support, Ridiculous!

 

Maybe we could all be a little more HUMAN and show some love, patience and understanding for one another. Geez, try and put yourself in their shoes.

 

Anyways, Thank you for giving me another opportunity to post a link to one worthy of starting.

https://www.gofundme...-in-crisis-help

 


  • 2


#7008 Major Outage - 09/21/18+ - Client Discussion

Posted by Brad on 23 September 2018 - 01:29 PM

Hello,

This is my plea to fellow loyal customers. I've been with MDD hosting and Michael since 2012 and I don't plan on going anywhere. I've been hosting websites for 18 years and I've experienced the worst the industry has to offer.

 

MDD Hosting has been amazing and finding a company with the level of support is very hard. Michael is a pillar of business ethics and I trust what he says. All that being said.

 

Being in IT, I know how a catastrophic event in business can be soul crushing. A little help from enough of us loyal customers can ease the pain and ensure a good reputable business and small business owner with a young family and employees to support can survive it. Please take a look and consider helping out.

 

If you can’t help Please Share.

If you want to be on the FundRaising Team click here. 

 

 

Please let me know if you have any questions.

 

Russ

 

Unbelievable.  While we sit in disarray... 


  • 2


#2751 [Completed] Fresco Server - Restoration Completed - File System Check Competed

Posted by MikeDVB on 21 June 2011 - 09:50 AM

I'd like to see a real back up plan in place.

We have a "real" backup plan. I'm not sure what makes you think otherwise.

-Fresco goes down due to whatever.
-You have one machine on the ready to restore all accounts from backups
-Done!

This is exactly what we did.

Congratulations for all the hard work you do is not enough for me, I think you do have a responsibility to put more effort into a back up plan to keep everyone's accounts up and running. You may say that yes u're not responsible for DDos attacks, but others I'm sure have lost clients due to something not their fault as well.

Ok, well nobody is forcing you to stay - if you wish to leave, by all means do what you feel is best.

So we just have to endure this again and again?

DDoS attacks - there's nothing we can do to stop/prevent those all we can do is mitigate them when they happen. As far as file system issues - this is only the second time we've ever had to run file system checks in over three and a half years in business. If that is "again and again" then I'm sorry. Beyond that, when the file system is corrupted, we don't have a choice but to run a file system check.

I'm not blaming you, but more should be put into finding a solution for these situations.

DDoS - We mitigate it. File System Corruption - we repair the file system. This is exactly what we've done.

If I were there, I surely would be heading up a project to do so. Nothing is impossible.

Heading up a project to do what? If you know a way to prevent DDoS attacks then you could very easily make yourself a millionaire selling whatever product/solution you come up with to prevent such attacks. And beyond that - if you come up with a way to prevent file system corruption on a kernel panic or other unexpected issue where disk caches may not be fully flushed to the disks - you should patent it.

I understand you are upset and/or frustrated but to be honest you really don't seem to have a solid grasp as to what exactly happened and what we did to bring everything back online. By all means, if you feel the grass is greener on the other side of the fence - feel free to find a new provider. This could happen to any provider on any server at any time - what is important about it is how that provider handles the issue. If there is something more you feel we should have done, by all means, do let me know what you think we should have done.

Nobody is perfect and we don't claim to be - we just do our absolute best to provide the best service and to resolve issues as quickly and efficiently as possible. We're not above and beyond taking constructive criticism and advice but if you're going to give advice, make sure you know what you're talking about before you do so. So I'm all ears, let me know what you think should have been done differently.
  • 2


#2601 [Completed] Fresco Server - Restoration Completed - File System Check Competed

Posted by mk88 on 20 June 2011 - 05:29 PM

for all you guys that say you can't afford to lose one day of business etc, you need to use dns failover.

We started using dnsmadeeasy. Just use their DNS service and for each site that you need online at all times, you setup failover service. So it pings your site every 5 minutes. If it goes down, it will automatically change the DNS to point to another server that you have the site mirrored on.

For example, i use two reseller accounts. This one and another with just copies of our important sites. When everything is down like now, my DNS points to the IP that is online, and everything is good. Using this setup i have had virtually no downtime for a long time.
  • 2


#7399 Major Outage - 09/21/18+ - Client Discussion

Posted by Bookproc on 26 September 2018 - 12:11 PM

I do NOT want to see "Service is down and will be fixed." I DO want to see a company being transparent and providing information along the way, including ETAs, if and when they are feasible, and explanations of changes in ETAs. In short, MDD provided all the information I wanted, perhaps more than I needed--but more is always better than less for me--and much much more than I could have expected in the circumstances. This was all deeply reassuring.

 

I have been with hosting providers that did the "It's down, it will be fixed" route, and *never again* is all I can say, because the corollary, though not expressed in precisely these terms, is generally "Now shut up and leave us alone, that's all we're going to tell you, and you're just going to have to wait it out without having a clue what's going on or how long it will take. Sucks to be you." I'm stuck with that kind of service with the cable company because their only competitor is just as bad, if not worse. Fortunately, MDD doesn't take that route, and I would never stay with a hosting provider that did

 

Different strokes for different folks.


  • 5