Just checking in to see how things are going with you and the crew, Mike. Hope you all managed to get some sleep this weekend and that the engineer who made the mistake is feeling a little better. Thanks again for all the updates.
- MDDHosting Forums
- → Viewing Profile: Posts: AMC4x4
AMC4x4Member Since 22 Apr 2015
Offline Last Active Jan 17 2019 10:39 AM
- Group Members
- Active Posts 24
- Profile Views 3,216
- Member Title Newbie
- Age Age Unknown
- Birthday Birthday Unknown
Posts I've Made
01 October 2018 - 03:24 PM
26 September 2018 - 11:45 AM
Just a note in response to "newbie" ... I, too, want to see MDDHosting grow and prosper. I've used their services many years and referred many people. But a couple of things:
1. This is 2018, not the mid-2000s when those generalized and generic terms were drafted.
2. This series of devastating technical errors is more akin to what happened a decade ago. This is 2018.
3. This is not so much about tech stuff, this is about company reputation ... how a person operates their business in 2018. Perception about a company's reputation is paramount, and Michael's lengthy emails, now reposted on industry websites, won't help MDDHosting's reputation. When I read them, I thought, "oh, no! Just say the service is down and it will be fixed."
4. Just this week, MDDHosting has disappeared from some rating services as a "top" site host.
5. Why not use your name?
Why so hostile? My name is Aaron Clow and I've been an MDD customer since maybe 2013 or so? I have two accounts with them - a corporate account for the company I work for and another account for my personal projects. If you care to look, many of us are "Newbie's" because MDD has had so few issues over the years that we haven't had to post here. "Newbie" is a classification, not my name. Also, when we sign up for a forum account, it doesn't put your name in as your username. You pick one. So I picked one.
If you prefer a hosting company that hides their activities behind a simple "service is down," and that's all the information you require when something untoward happens, I'm sure there are plenty of cheap places you can go for that kind of service. That's not how I do business. I'm forthright with people I speak to, and the reason I'm still with MDD is precisely because they do business this way. Look at how many people are still here and who say they aren't leaving. Why do you think that is?
So what's your name? Do people call you "Dehenderson?"
26 September 2018 - 11:07 AM
To the person who started a crowd funding initiative on behalf of MDDHosting, just let me express that MDDHosting is not the victim, all of the company's clients are. All of the company's clients who trusted that MDDHosting would professionally manage their websites were injured in one way or the other. If there is, indeed, a crowd funding initiative, then the recipients of any funds should be MDDHosting's customers, the people who suffered and were inconvenienced. Here's why -- what happened was not an "act of God." It was not a server failure or a datacenter fire.
What happened was human incompetence ... a inexcusable human mistake ... a self-inflicted error ... or, a "man made disaster" as a lawyer friend calls it. It should not have happened. And, liability (responsibility) is with MDDHosting.
Perhaps even worse is the fact that MDDHosting’s backup system - which would have enabled expeditious recovery - had been deactivated …. turned OFF. Another “man made disaster.”
You and I learned about this through at least two extraordinarily lengthy emails from Michael Denney, including details on the incorrect code entered that deleted everything. As someone who respects Michael, I wish he had not spent time writing those emails and more time on finding a fix. Additionally, those emails expose the company to untold legal exposure at this point.
As for our sites, we are discussing whether to throw in the towel. Our primary site is BoomerCafe.com, a baby boomer story sharing initiative that was launched in 1999. Nearly 20 years of equity has been wiped out in less than a week. I hope Michael and MDDHosting will find a professional and responsible way to make us … ALL of us … whole with our respective damages due to this disaster that was clearly … CLEARLY … the result of human failure.
1) What happened was actually a series of events, not a single event. It might have seemed like a single event, but it wasn't *backups* that were offline, it was snapshots. The backup server was working, slow as it was, and thank god for that. Otherwise many of us would have been well and truly $crewed.
2) I agree about the GoFundMe. While many of us are still fans of Michael and MDD, this has cost us as well as them, and while I'm keeping most of my sites with MDD, I'm not donating to the GoFundMe because we are all out something here. I spent money to move one of my sites elsewhere that I had an UpdraftPlus backup for. Regular backups and backup testing are a must for anyone who maintains content and deals with IT. I only had those for some of my sites. That's on me. If I had regular cPanel backups, I would have been back online within 24 hours at most. Again, shared responsibility. I believe the ToS say you are responsible for your own backups.
3) I believe the ToS covers Mike and MDD from "untold legal exposure" because at the end of the day, YOU are responsible for backing up your data.
4) Writing those letters did not take away from time spent on restoration. There was a well-documented bottleneck with the backup server that gave Mike and his crew time to answer tickets, keep us updated, etc. I'm sure if they could have sped that copying process along, they would have been happy to have spent time on restoration rather than taking the time to patiently respond to all our questions.
5) I see your site is up, with content as recent as the 18th. How do you figure "Nearly 20 years of equity has been wiped out in less than a week." Just curious
23 September 2018 - 07:40 PM
At least you have a process now and can see the progress! Thanks for keeping us up to date.
23 September 2018 - 05:11 PM
@Mike thanks for validating. The validation email was never received anywhere (gmail, domain registered with you, etc).
I was able to get my account partially restored manually by having you create a 'new' account.
I then downloaded the backup file from domain cpanel\full account backups\generate download. This produced a rather large (2.5gb) .gz file which I then restored through ssh to the home directory. Files in the homedir had to be relocated, but mostly everything else appears to be in the right place.
What's missing are the dns records, subdomains, databases and some others.
Jetbackup documentation references a restore button that should be present next to the generate backup button. However it's not there. How can I fully restore the account myself, including the above missing items?
My concern is if the automated restore runs, new emails which have been received since the crash will be wiped from the mail storage. Some have been downloaded, but some clients also use imap.
Mike, I'm wondering if part of the sudden slowdowns you might have experienced from the backup server were a result of some of us grabbing full account backups while you were also trying to restore? I did this before you mentioned any speed issues on your part, but the backup did take a long time. It looks like others had the same idea as soon as they regained access to your account. You said you had little vision into what the backup server was doing, but I would think this would have been part of the issue - you'd be humming along, and then probably one of us started grabbing a backup, causing a slowdown for you. I wonder if there's a way to turn off client generated backups until you get the data off that machine? Just a thought if you hadn't thought of it already.