Major Outage - 09/21/18+ - Client Discussion

[Michael D.]

now god forbid but if this happens again im sure alot of us is going to leave. that just means pure carelessness / laziness

 

1. It is absolutely not going to happen again.
2. Even if it did happen again - the changes we've made will allow us to recover the entire network in minutes - without actually having to do any 'restores' or IP changes or anything of the like.

This is due to just one change we've made - and there are dozens - some already implemented and some pending full restoration so we can focus on them.

[jsolo]

 

As I told you in the ticket - initially you wanted it overwritten and then you asked for it not to be - so you ended up on both lists. Admittedly we have to go out of our way to overwrite your account - so when we saw it on that list we did it. Did we ****** up? Yes. It's worth noting that we aren't restoring your accounts from our support ticket system - but from a restoration interface on the server itself that does not in any way reference the ticket system. If we were to reference the ticket system for every restore - we'd have only a few hundred accounts online at this time instead of many thousands.

 

Did we mess up? Absolutely - and I'm sorry for that.

 

http://www.screen-shot.net/2018-09-25_20-41-27.png

 

Chronologically, yes, initially I did want it restored until I discovered the backup file. Once that backup file was restored I even confirmed in the ticket that the account won't be touched when the s3 restore happens. No actual confirmation was given but I can only hope that tickets gets read thoroughly.

 

If you've got two lists, isn't it prudent to make sure someone isn't on both at the same time.

 

I'm stepping away from this... Frustrated enough as it is.

[Michael D.]

 

Chronologically, yes, initially I did want it restored until I discovered the backup file. Once that backup file was restored I even confirmed in the ticket that the account won't be touched when the s3 restore happens. No actual confirmation was given but I can only hope that tickets gets read thoroughly.

 

If you've got two lists, isn't it prudent to make sure someone isn't on both at the same time.

 

I'm stepping away from this... Frustrated enough as it is.

Yes, it's prudent to make sure nobody is on the list twice and that's the reason I checked for other duplicates after you brought this to our attention. While I can't undo what happened in your case, I can learn from it and make sure it doesn't happen again.

[Stahlrosen]

We are back up and running fine on s3. Thank you team MDD for all your hard work.

[Michael D.]

For those worried that something like this could happen again.

 

We have already enabled snapshots on our storage cluster. We're doing one snapshot every hour and keeping them for 10 hours.

 

So from a hypothetical standpoint - let's say that this did manage to happen again. We would simply mount a snapshot from before the incident - within the hour before - and boot everything back up.

 

Total downtime would be - ~5 minutes - for the whole network. Would there be any data loss? Possibly anything written in the preceding hour or less - but nothing compared to the losses of a multi-day outage.

 

It would look literally like we just shut everything down and booted it back up. No 'restorations', no lost emails, nothing. There's a great chance almost nobody would even notice.

 

This is something that our storage vendor, StorPool, set up for us immediately upon seeing what had happened. They actually apologized that it was not already set up and said that as a result of our disaster they are going to make sure that it is a default behavior that has to be actively disabled rather than the other way around.

 

Even with these snapshots and as powerful as they are - we are still going to overhaul our backup servers. We have identified the issues with the present setup that caused restorations to be so slow and already have fixes for those issues planned for once we are fully online and all of our clients are taken care of.

 

Snapshots are a very powerful tool against data loss and corruption. We actually used them a couple of times on our old storage platform, the Nimble CS500, to recover data on servers when clients made big mistakes themselves.

[bobptz]

@Michael Denney

 

Restoring from a snapshot taken 45 min ago, is not like a simple reboot. You will loose all emails from the last 45 min.

[AMGill29]

I was astonished to read the following statement on the locked outage forum today: "If anybody has a backup of their own - we can use that to get you online immediately. " I didn't know that. I wish it had been said in the MDDHosting emails about the outage. I meticulously keep a backup. Well, I'm on S3 and should be restored by late tonight or very early a.m. tomorrow anyway, so I don't see the point of sending in my backup now. It would just add unnecessary work to an already overburdened staff. And I still have confidence in MDDHosting. But talk about rubbing salt in a wound....

Same here. I didn't read this till later and assumed it meant a Cpanel backup. I have files and database and by then it was too late for me too. I spent the whole weekend at my computer up late restoring as many sites as I could on another host to get my clients sites live. This would have saved me so much hassle and a host of other DNS resolving issues if this was made clear to me. I could care less about the money I spent just about the stress and now some of my clients have lost trust in me. I have bragged about MDD to everyone that signed up with me. I understand and feel bad about the domino of events that happened - mostly out of MDDs control but this one was a hard pill to swallow for me as well.

[SarisIsop]

All up and running on S5

 

Have lost a few posts on our forum, but it's no big deal and we are just pleased to be back on-line.

 

I would like to say a big thank you to Michael and all the staff and helpers at MDD for getting us out of this. No it shouldn't have happened, but it did and you all rolled up your sleeves on got on with.

 

Michael should be commended for transparency and honesty to his clients, it takes guts to take it on the chin but he has proven that he will always put his customers first.

 

I also want to shout out to the person that made this happen. Don't dwell on it, it's the past and you can not change that. Dust yourself off, hold your head up high and carry on your journey into the future.

 

We are all to blame and I for one will be taking regular cPanel backups, had I have done that I would have been back on-line within a couple of hours instead of waiting till today. Lesson learnt.

 

I will not be leaving MDD. I'm only a small fish but they have always looked after me, sometimes far beyond what I pay them for. I also will not be requesting any refund/compensation as I have lost no money over this and I feel you will be losing enough.

 

I know all of you at MDD have still a lot of work ahead of you and yes you will undoubtedly be on the big side of losing, but this episode has made you stronger and I still have full faith in you.

 

Sorry to those of you that have been posting your anger especially Avatar for telling me I will just have to "Be Annoyed" I do understand as I once ran my own business, but I just wanted the original thread to be updates and not questions or complaints. Anyway Michael did eventually split the thread and lock it, something I would recommend doing in future.

 

Good luck to everyone whichever way you have decided to go, and thank you MDD.

[ninjaclan]

Any ETA for S4 full restoration?

[rnoar]

Please - ETA on S4 accounts fully restored.

[SarisIsop]

I'm guessing things will take a while to settle down. ?

 

Posts made on my forum since being back on-line have suddenly gone, I have closed the forum down for now.

 

Would anyone know when things will start getting saved again.

 

Thank you.

[ADC]

Not off hand - does it require any custom ports?

This was my mistake. I forgot that Friday's orders were missing from the database, so new orders were assigned their order numbers. Fixed.

 

 

@Mike, unless you're going to be offering some serious credits for the outage I have a hard time justifying a reason to stay.

 

Well, good luck to you if you decide to leave. I've had nothing but trouble with every other host I've been with- hacks, outages... and they frankly didn't give a rip if my site was up or not. I'll be more careful about my backups from now on, but I don't recall being down once prior since hosting with MDD. This kind of a thing was a yearly occurrence with others.

[sf2099]

For whatever is worth to people considering leaving MDD, I would like to say that I will be staying with MDD.

 

Although I don't share the fanboysim from few individuals here, I know that Mike and his team have been solid since becoming their customer in Jan 2013.

 

Yes, no doubt about it. This was a big, self-inflicted f***up. BUT, Mike and his team stepped up and owned it and I believe that deserves some credit.

 

And I like his plans on minimizing the risk of these types of catastrophic events from happening again in the future. I would like to hear more about backup testings in future.

 

As a small business owner myself, I do like supporting other small businesses. So as long as Mike delivers on his promises, he has my support.

 

Lastly, what happened here does not absolve myself from being more diligent either. Here is a screenshot of what Mike told me back in 2013 before I became his customer. A good advice, still in 2018!

 

 

[Dehenderson]

Removed by Request

Edited September 26, 2018 by MikeDVB
Removed at the request of Dehenderson

[smikwily]

Can we get a status update on S4, please?

[LShoe]

I had restored from my own backups and my site was up, but I gave the go ahead to overwrite this with the S5 server backup. My site is currently down (with the terrible "Sorry" message". I don't know when it went down, but is this a very short outage while the restore process is actively writing to my site, or is this a long outage while you write the entire contents of the server and then afterwards gradually turn on the accounts?

[ithelpme2]

@MDD Hosting. Hello, I'm very familiar with Nimble snapshots. I work with Nimble storage on a daily basis at work. I'm the Systems Engineer at work, website hobbyists outside of work. You mentioned that you keep up to 10 hours worth of snapshots? Unless you're pressed for space, I think 10 hours is nothing. Our file servers at work, we snapshot hourly and keep that for years. It protects us from unknown vulnerabilities in which we may need to get data past the unexpected time frame.

[digibread]

A few things I've learned in the 18 years of hosting websites:

 

1. Have your own backups stored offsite.

- I back up databases daily and file system weekly for all sites.

2. Don't be a cheap hosting service. The lower your prices the lower the quality of customers.

- I only offer a managed hosting service for a premium price (1000% higher than the cheap hosts). Fewer clients but they are the best.

- I offer a 50% discount to non-profits (and believe it or not they were the squeakiest wheels during this outage.) so you get my point.

3. Don't host email unless you want to hate life and lose money. Refer them to Google Hosted Apps or Office 365, and if you must resell email try GoDaddy Workspace.

- I host email for only 3 clients, on MDD and charge for it separately. It is by far and away, what I get most support tickets on. My customers pay extra because they want to get me on the phone. None of them complained throughout this event even being down for 4 days.

- I also charge separately (hourly) for DNS service and support, SSL service and support, as well as helping setup 3rd party email and supporting it ongoing.

4. In your pre-sales process make it clear that outages happen (this is technology) and you'll be there for them, that you have a plan and you'll always be honest and forthright. (if they don't do business with you because of this. GOOD)

5. If you build websites, build them locally and use a versioning system like Git (my favorite, especially with BitBucket), SVN, etc.. Need a LAMP stack look at Laragon (my favorite), Serverpress Desktop Server, or Local by Flywheel, there is also XAMPP and WAMPP

6. When you find a good hosting provider (like MDD) be loyal and stick with them.

- I've been an administrator and made mistakes equal to what happened here, albeit on a smaller scale of customers but nonetheless, it sucks and I always had great friends, family and customers that stuck by me and encouraged me through the crap storm.

- If you are hosting websites and put the onus on the security of your data on ANY third party provider you're a fool and need to get out.

 

Doing the above mitigated my stress to a minimum and my loss to zero through this. I understand that problems like these are relative, it hurts and it's never fun. However when I compare MDD to my prior experiences they have been stellar. Yes they had dramatic process and system failures but don't forget, their last line of defense worked. Not only that but they never give up trying new methods to speed up the process and I'm ultra impressed by the fact that things will wrap up soon.

 

MDD Hosting I'm not going anywhere.

[Djqueball]

Can we get a status update on S4, please?

https://forums.mddhosting.com/topic/1582-major-outage-092118-09242018/page-5

[smikwily]

https://forums.mddhosting.com/topic/1582-major-outage-092118-09242018/page-5

Thanks, but that was posted 4 minutes after I asked for a status update. That post wasn't available when I posted.

[digibread]

 

To the person who started a crowd funding initiative on behalf of MDDHosting, just let me express that MDDHosting is not the victim, all of the company's clients are. All of the company's clients who trusted that MDDHosting would professionally manage their websites were injured in one way or the other. If there is, indeed, a crowd funding initiative, then the recipients of any funds should be MDDHosting's customers, the people who suffered and were inconvenienced. Here's why -- what happened was not an "act of God." It was not a server failure or a datacenter fire.

 

What happened was human incompetence ... a inexcusable human mistake ... a self-inflicted error ... or, a "man made disaster" as a lawyer friend calls it. It should not have happened. And, liability (responsibility) is with MDDHosting.

 

Perhaps even worse is the fact that MDDHosting’s backup system - which would have enabled expeditious recovery - had been deactivated …. turned OFF. Another “man made disaster.”

 

You and I learned about this through at least two extraordinarily lengthy emails from Michael Denney, including details on the incorrect code entered that deleted everything. As someone who respects Michael, I wish he had not spent time writing those emails and more time on finding a fix. Additionally, those emails expose the company to untold legal exposure at this point.

 

As for our sites, we are discussing whether to throw in the towel. Our primary site is BoomerCafe.com, a baby boomer story sharing initiative that was launched in 1999. Nearly 20 years of equity has been wiped out in less than a week. I hope Michael and MDDHosting will find a professional and responsible way to make us … ALL of us … whole with our respective damages due to this disaster that was clearly … CLEARLY … the result of human failure.

 

 

Sir, I started the fundraiser and I don't mean any disrespect to you, but you (and anyone else thinking like you) clearly did not read your end user agreement when you signed up. Your Data and Access to it, is Your Responsibility! That goes with any hosting provider out there. Your loss in equity is your fault and yours alone. There are so many ways (for site owners, hosting resellers, etc.) to mitigate what all of us experienced here that there is no excuse. The final responsibility will always and does now stop with you.

 

Human error will always play into anything we do as businesses and most companies could give a $h1t about their customers. MDD is not like that at all and Michael's transparency is refreshing and much appreciated. MDD is a small business with an exceptional owner with a young family and responsibilities to employees. They clearly have responsibilities to their customers and have performed to the best of their abilities given the situation.

 

I had over 32 customers down for 4 days, including a National Food Chain and not one single loss or complaint like yours. You see, even if MDD was a terrible service provider and didn't have any backup I could have quickly setup new hosting and them all back online within a 24 and the most critical within a few. However, each one trusted me that MDD would come through and they did.

 

You are correct in this: MDD is not a victim, but sir, neither are you! The fundraisers that piss me off are the ones for political douche bags. And they get untold amounts of support, Ridiculous!

 

Maybe we could all be a little more HUMAN and show some love, patience and understanding for one another. Geez, try and put yourself in their shoes.

 

Anyways, Thank you for giving me another opportunity to post a link to one worthy of starting.

https://www.gofundme.com/fellow-tech-in-crisis-help

 

[AMC4x4]

 

To the person who started a crowd funding initiative on behalf of MDDHosting, just let me express that MDDHosting is not the victim, all of the company's clients are. All of the company's clients who trusted that MDDHosting would professionally manage their websites were injured in one way or the other. If there is, indeed, a crowd funding initiative, then the recipients of any funds should be MDDHosting's customers, the people who suffered and were inconvenienced. Here's why -- what happened was not an "act of God." It was not a server failure or a datacenter fire.

 

What happened was human incompetence ... a inexcusable human mistake ... a self-inflicted error ... or, a "man made disaster" as a lawyer friend calls it. It should not have happened. And, liability (responsibility) is with MDDHosting.

 

Perhaps even worse is the fact that MDDHosting’s backup system - which would have enabled expeditious recovery - had been deactivated …. turned OFF. Another “man made disaster.”

 

You and I learned about this through at least two extraordinarily lengthy emails from Michael Denney, including details on the incorrect code entered that deleted everything. As someone who respects Michael, I wish he had not spent time writing those emails and more time on finding a fix. Additionally, those emails expose the company to untold legal exposure at this point.

 

As for our sites, we are discussing whether to throw in the towel. Our primary site is BoomerCafe.com, a baby boomer story sharing initiative that was launched in 1999. Nearly 20 years of equity has been wiped out in less than a week. I hope Michael and MDDHosting will find a professional and responsible way to make us … ALL of us … whole with our respective damages due to this disaster that was clearly … CLEARLY … the result of human failure.

 

 

1) What happened was actually a series of events, not a single event. It might have seemed like a single event, but it wasn't *backups* that were offline, it was snapshots. The backup server was working, slow as it was, and thank god for that. Otherwise many of us would have been well and truly $crewed.

2) I agree about the GoFundMe. While many of us are still fans of Michael and MDD, this has cost us as well as them, and while I'm keeping most of my sites with MDD, I'm not donating to the GoFundMe because we are all out something here. I spent money to move one of my sites elsewhere that I had an UpdraftPlus backup for. Regular backups and backup testing are a must for anyone who maintains content and deals with IT. I only had those for some of my sites. That's on me. If I had regular cPanel backups, I would have been back online within 24 hours at most. Again, shared responsibility. I believe the ToS say you are responsible for your own backups.

3) I believe the ToS covers Mike and MDD from "untold legal exposure" because at the end of the day, YOU are responsible for backing up your data.

4) Writing those letters did not take away from time spent on restoration. There was a well-documented bottleneck with the backup server that gave Mike and his crew time to answer tickets, keep us updated, etc. I'm sure if they could have sped that copying process along, they would have been happy to have spent time on restoration rather than taking the time to patiently respond to all our questions.

5) I see your site is up, with content as recent as the 18th. How do you figure "Nearly 20 years of equity has been wiped out in less than a week." Just curious

[Dehenderson]

Just a note in response to "newbie" ... I, too, want to see MDDHosting grow and prosper. I've used their services many years and referred many people. But a couple of things:

 

1. This is 2018, not the mid-2000s when those generalized and generic terms were drafted.

 

2. This series of devastating technical errors is more akin to what happened a decade ago. This is 2018.

 

3. This is not so much about tech stuff, this is about company reputation ... how a person operates their business in 2018. Perception about a company's reputation is paramount, and Michael's lengthy emails, now reposted on industry websites, won't help MDDHosting's reputation. When I read them, I thought, "oh, no! Just say the service is down and it will be fixed."

 

4. Just this week, MDDHosting has disappeared from some rating services as a "top" site host.

 

5. Why not use your name?

[AMC4x4]

Just a note in response to "newbie" ... I, too, want to see MDDHosting grow and prosper. I've used their services many years and referred many people. But a couple of things:

 

1. This is 2018, not the mid-2000s when those generalized and generic terms were drafted.

 

2. This series of devastating technical errors is more akin to what happened a decade ago. This is 2018.

 

3. This is not so much about tech stuff, this is about company reputation ... how a person operates their business in 2018. Perception about a company's reputation is paramount, and Michael's lengthy emails, now reposted on industry websites, won't help MDDHosting's reputation. When I read them, I thought, "oh, no! Just say the service is down and it will be fixed."

 

4. Just this week, MDDHosting has disappeared from some rating services as a "top" site host.

 

5. Why not use your name?

 

Why so hostile? My name is Aaron Clow and I've been an MDD customer since maybe 2013 or so? I have two accounts with them - a corporate account for the company I work for and another account for my personal projects. If you care to look, many of us are "Newbie's" because MDD has had so few issues over the years that we haven't had to post here. "Newbie" is a classification, not my name. Also, when we sign up for a forum account, it doesn't put your name in as your username. You pick one. So I picked one.

 

If you prefer a hosting company that hides their activities behind a simple "service is down," and that's all the information you require when something untoward happens, I'm sure there are plenty of cheap places you can go for that kind of service. That's not how I do business. I'm forthright with people I speak to, and the reason I'm still with MDD is precisely because they do business this way. Look at how many people are still here and who say they aren't leaving. Why do you think that is?

 

So what's your name? Do people call you "Dehenderson?"

 

Cheers

[Michael D.]

@Michael Denney

 

Restoring from a snapshot taken 45 min ago, is not like a simple reboot. You will loose all emails from the last 45 min.

I am talking about downtime incurred, I did say that some data could be lost from up to the last hour.