AMC4x4

Just checking in to see how things are going with you and the crew, Mike. Hope you all managed to get some sleep this weekend and that the engineer who made the mistake is feeling a little better. Thanks again for all the updates.

Why so hostile? My name is Aaron Clow and I've been an MDD customer since maybe 2013 or so? I have two accounts with them - a corporate account for the company I work for and another account for my personal projects. If you care to look, many of us are "Newbie's" because MDD has had so few issues over the years that we haven't had to post here. "Newbie" is a classification, not my name. Also, when we sign up for a forum account, it doesn't put your name in as your username. You pick one. So I picked one. If you prefer a hosting company that hides their activities behind a simple "service is down," and that's all the information you require when something untoward happens, I'm sure there are plenty of cheap places you can go for that kind of service. That's not how I do business. I'm forthright with people I speak to, and the reason I'm still with MDD is precisely because they do business this way. Look at how many people are still here and who say they aren't leaving. Why do you think that is? So what's your name? Do people call you "Dehenderson?" Cheers

1) What happened was actually a series of events, not a single event. It might have seemed like a single event, but it wasn't *backups* that were offline, it was snapshots. The backup server was working, slow as it was, and thank god for that. Otherwise many of us would have been well and truly $crewed. 2) I agree about the GoFundMe. While many of us are still fans of Michael and MDD, this has cost us as well as them, and while I'm keeping most of my sites with MDD, I'm not donating to the GoFundMe because we are all out something here. I spent money to move one of my sites elsewhere that I had an UpdraftPlus backup for. Regular backups and backup testing are a must for anyone who maintains content and deals with IT. I only had those for some of my sites. That's on me. If I had regular cPanel backups, I would have been back online within 24 hours at most. Again, shared responsibility. I believe the ToS say you are responsible for your own backups. 3) I believe the ToS covers Mike and MDD from "untold legal exposure" because at the end of the day, YOU are responsible for backing up your data. 4) Writing those letters did not take away from time spent on restoration. There was a well-documented bottleneck with the backup server that gave Mike and his crew time to answer tickets, keep us updated, etc. I'm sure if they could have sped that copying process along, they would have been happy to have spent time on restoration rather than taking the time to patiently respond to all our questions. 5) I see your site is up, with content as recent as the 18th. How do you figure "Nearly 20 years of equity has been wiped out in less than a week." Just curious

At least you have a process now and can see the progress! Thanks for keeping us up to date.

Mike, I'm wondering if part of the sudden slowdowns you might have experienced from the backup server were a result of some of us grabbing full account backups while you were also trying to restore? I did this before you mentioned any speed issues on your part, but the backup did take a long time. It looks like others had the same idea as soon as they regained access to your account. You said you had little vision into what the backup server was doing, but I would think this would have been part of the issue - you'd be humming along, and then probably one of us started grabbing a backup, causing a slowdown for you. I wonder if there's a way to turn off client generated backups until you get the data off that machine? Just a thought if you hadn't thought of it already.

Yeah, iTunes scrapes my blog and posts a podcast and they ended up removing it, I guess because the SSL version of the feed URL seems to point to some site about Emiliya. Maybe they thought I had been hacked. I moved that blog over to a new hosting account and I hope I haven't lost all my reviews and ratings. The blog/podcast doesn't get a lot of traffic, but I worked hard to earn what little I have.

One of my sites was restored, but the SSL certificate is showing as invalid and it's redirecting to the MDD forums (if people click to accept the improper certificate), so perhaps clients will know that it's a hosting issue. Hopefully they don't think we've been hacked.

I bet Michael would differ with you on "patient lot," as it seems he's been getting lots of nasty messages. People are rightfully frustrated. Thing is, this company is way more transparent than any other company I've ever dealt with, and if there is any forgiving or patience on the part of customers, I'd say that's the only reason. We do tolerate a bit of downtime for the decent prices and excellent service, and it's because we know the MDD team is solid. They're the types of people we'd hire if we were in charge of hiring. Mistakes happen, and I'm sure they will learn from this.

I'm just grateful that your last resort backup actually had valid data and could be used. 96 hours I can deal with, but loss of data would be catastrophic for many of us. I'm sure we've all learned some things. For instance, I kept my backups on my MDD account. That was stupid on my part, but I thought your triple backups would be enough. This has been a huge wake up call for me to implement a regular personal backup routine that I should have been doing all along. I got complacent. Thanks again for all your hard work getting us back up and running, for the frequent updates, and for the transparency. -- still a customer for the long term

As always, thanks for the detailed report. Great work getting to the bottom of it. Hope the rest of your night is uneventful.

Have you noticed any reduction in performance due to the patches? Just curious. Thanks.

Thank you for the RCA. As always, the transparency and these RCA's are a big reason why I stay here and re-signed for three more years. Thanks again!

@Dehenderson - Perhaps it's an issue where Storpool can log into MDD's storage and monitor the services? We have some Infinidat boxes in our lab, and they are always contacting us to see when they can schedule something or other. Either way, I'm curious as well.

OK, thanks.

Hi Mike, I gather you're still working on P1? Just had another brief outage. Thanks.