Jump to content
MDDHosting Forums

Major Outage - 09/21/18+ - Client Discussion


KevinD872

Recommended Posts

I can't speak for everyone, but while I'm not too happy about this, I can understand and empathize with your side of this situation.

 

That said, I do have one question: How recent is the data backup on your "last resort disaster recovery" server(s)?

Link to comment
Share on other sites

I can't speak for everyone, but while I'm not too happy about this, I can understand and empathize with your side of this situation.

 

That said, I do have one question: How recent is the data backup on your "last resort disaster recovery" server(s)?

Most are only about 14 hours before the outage, one or two servers are 48 hours or so.

  • Upvote 1
Link to comment
Share on other sites

Most are only about 14 hours before the outage, one or two servers are 48 hours or so.

 

One last question: What will happen with any emails we should have received between the problem occurring and when our sites come back online?

Link to comment
Share on other sites

 

One last question: What will happen with any emails we should have received between the problem occurring and when our sites come back online?

 

It's up to the sending server to queue them and attempt delivery until the mail server comes back online. I wish there were something we could do about it beyond that.

 

Well mistakes happen but a recovery plan should have covered that. Quickly.

 

Snapshots were disabled, some backups are 14 hours some are 2 days.

 

Not a good plan guys. Not a good plan at all. In 2018.

 

It's not that we had one thing go wrong - we had one thing, and then another, and then another, and then another. We'll explain it all in the official RFO - but needless to say there should have been no situation where we actually had to restore from this particular disaster recovery system.

 

Link to comment
Share on other sites

It's not that we had one thing go wrong - we had one thing, and then another, and then another, and then another. We'll explain it all in the official RFO - but needless to say there should have been no situation where we actually had to restore from this particular disaster recovery system.

 

 

I'm just grateful that your last resort backup actually had valid data and could be used. 96 hours I can deal with, but loss of data would be catastrophic for many of us. I'm sure we've all learned some things. For instance, I kept my backups on my MDD account. That was stupid on my part, but I thought your triple backups would be enough. This has been a huge wake up call for me to implement a regular personal backup routine that I should have been doing all along. I got complacent.

 

Thanks again for all your hard work getting us back up and running, for the frequent updates, and for the transparency.

 

-- still a customer for the long term

Link to comment
Share on other sites

I guess MDD clients are mostly a patient lot. But if you are not making much or any money from your sites, maybe it's "ok".

But the fact is, just one day of downtime will cost (some) clients a lot of money, many times more than the credit refunds that MDD would ever give back as "compensation".

 

The worse case scenario (hope I'm wrong) is 3+ days of outage (it's extremely bad if you are earning from your sites).

 

I suspect when Michael said he may not get sleep for 72 hours, this is what he had in mind. :(

Link to comment
Share on other sites

I guess MDD clients are mostly a patient lot. But if you are not making much or any money from your sites, maybe it's "ok".

 

But the fact is, just one day of downtime will cost (some) clients a lot of money, many times more than the credit refunds that MDD would ever give back as "compensation".

 

The worse case scenario (hope I'm wrong) is 3+ days of outage (it's extremely bad if you are earning from your sites).

 

I suspect when Michael said he may not get sleep for 72 hours, this is what he had in mind. :(

I hope for the best but plan for the worst.

 

We tried numerous things to speed this process up, which I will elaborate later as I'm short on time right now, but none of them worked.

 

There was one or two we could have tried that would have put the last set of data at risk - and we decided that it was better to have slower restorations than to possibly lose the data.

Link to comment
Share on other sites

I guess MDD clients are mostly a patient lot. But if you are not making much or any money from your sites, maybe it's "ok".

 

But the fact is, just one day of downtime will cost (some) clients a lot of money, many times more than the credit refunds that MDD would ever give back as "compensation".

 

The worse case scenario (hope I'm wrong) is 3+ days of outage (it's extremely bad if you are earning from your sites).

 

I suspect when Michael said he may not get sleep for 72 hours, this is what he had in mind. :(

 

I bet Michael would differ with you on "patient lot," as it seems he's been getting lots of nasty messages. People are rightfully frustrated. Thing is, this company is way more transparent than any other company I've ever dealt with, and if there is any forgiving or patience on the part of customers, I'd say that's the only reason. We do tolerate a bit of downtime for the decent prices and excellent service, and it's because we know the MDD team is solid. They're the types of people we'd hire if we were in charge of hiring. Mistakes happen, and I'm sure they will learn from this.

Link to comment
Share on other sites

@AMC4x4 well said. If it wasn't for the transparency, I'd be looking for a new host, but I'm not.

 

Human beings make mistakes, and it's how they respond to those mistakes that matters. It's absolutely clear that the MDD team are doing their level best in a really horrible situation. They're the kind of guys I want on my team.

 

For those complaints I've seen on Twitter - and I'm sure there's much worse on support tickets - all of us have had hair raising days in business, and if you haven't, you just haven't been in business long enough yet. Follow MDD's example, be transparent and honest. If your clients aren't willing to support you even though you're not perfect, they're not the clients you want.

 

@MikeDVB make sure your team get some sleep (and you too!). Exhausted people make bigger mistakes.

  • Upvote 1
Link to comment
Share on other sites

Sometimes stuff happens. We learn from it, and get better from the experience.

 

How a company responds in a crisis situation like this says the most, and you guys are doing a STELLAR job. The transparency and dedication is greatly appreciated, and I won't nag or delay you by putting in a ticket. Everything I need to know was posted in these updates.

 

Thanks to the MDD staff, in advance, for another job well done.

 

Hopefully you'll get some well deserved sleep soon!

 

Mike

  • Upvote 1
Link to comment
Share on other sites

Just want to encourage Mike and the team to keep up the good work.

 

I have been with MDD for 8 years and Mike always has and still is honest with his customers. Yes it's annoying and I too am getting a lot of flack form my forum members, but at the end of the day it's best to just let MDD get on with it.

 

I'm lucky as I don't want to make money from my websites, but I do run a forum for people with a health problem and a lot of those people NEED my forum so it does effect us all in different ways.

 

No it shouldn't happen, but it has so now we just have to leave them to get on with it and get us all back on-line. Which I'm sure after my dealings with them that they will as soon as they can.

 

We can all learn from this and it's not till it goes wrong that we appreciate what we have. I still however feel safe with MDD and Mike should be proud of his honesty to his customers, I doubt many other owners would be that honest.

 

I still have full faith in Mike and his staff, I actually feel sorry for the person that pushed the wrong button so to speak. So knuckle down MDD and do your stuff, we are in your hands.

Link to comment
Share on other sites

I would prefer to see something different than IP address has changed, server misconfiguration or the site may have moved when trying to reach my site. Why not simply telling hosting provider is experiencing difficulties. Now all my clients think I caused a downtime myself and to be honest I am tired of phone calls my site is down and what am I going to do to make it alive, etc.

Just my 2p.

Link to comment
Share on other sites

I would prefer to see something different than IP address has changed, server misconfiguration or the site may have moved when trying to reach my site. Why not simply telling hosting provider is experiencing difficulties. Now all my clients think I caused a downtime myself and to be honest I am tired of phone calls my site is down and what am I going to do to make it alive, etc.

Just my 2p.

We have a fair bit of clients ourselves and I understand fully about being tired of hearing about sites being down. I mean it's not like you want them offline, right? Neither do we.

 

I wish there was a faster way for us to get everything back online.

 

At the end of the day though it's our responsibility to support our customers. I just wish we weren't getting 5 new tickets every 3 seconds - it's an untenable amount of incoming tickets to respond to.

Link to comment
Share on other sites

Our servers are exceptionally fast and can move data at gigabytes per second - it's our remote backup server that's slow and can't keep up.

 

I really honestly don't know how long this is going to take - that's the primary reason I keep not giving an ETA.

 

As it stands there are still thousands upon thousands of accounts to restore. I'd like to say I'll get to sleep within the next few days but honesty I really don't know.

Link to comment
Share on other sites

I would prefer to see something different than IP address has changed, server misconfiguration or the site may have moved when trying to reach my site. Why not simply telling hosting provider is experiencing difficulties. Now all my clients think I caused a downtime myself and to be honest I am tired of phone calls my site is down and what am I going to do to make it alive, etc.

Just my 2p.

 

One of my sites was restored, but the SSL certificate is showing as invalid and it's redirecting to the MDD forums (if people click to accept the improper certificate), so perhaps clients will know that it's a hosting issue. Hopefully they don't think we've been hacked.

Link to comment
Share on other sites

Can you please clarify the best/easiest way for us to check if an account has been restored without opening unnecessary support tickets?

Right now I am assuming that my account(s) has been not restored yet if I get an authentication error while trying to login to cPanel.

Thank you for being so open and transparent, much appreciated.

Link to comment
Share on other sites

Please understand, no gripe, just tinfoil hat, okay? 5:12PM MDT and my Wordpress blog (plus two addon sites) are still down. I have full backups for the addons, no problem. For the Wordpress database, blog entries dating back 15 years, I’m nervous that (*if* they’re restored) I shouldn’t also “backup” a blind copy to my “secret” Wordpress.com site. Else I can’t imagine how to maintain an off-host backup.

 

Ah, well, thanks so much for keeping us up to date pards. I’m sure you know it’s painful for us and I know it’s painful as well for you. I’m sorry my email of sympathy and gratitude generated a ticket. Unintended. Live long and prosper.

Link to comment
Share on other sites

Was a little reassuring for me to read of other's sites getting back online, so I'll add that my critical site is back up now (where customers download auxiliary content for things they have already purchased). My other site (where customers can buy stuff) isn't as much an issue for me in comparison because it's not part of content people have already purchased, so I don't mind waiting on that one. Best Regards all.

Link to comment
Share on other sites

Please understand, no gripe, just tinfoil hat, okay? 5:12PM MDT and my Wordpress blog (plus two addon sites) are still down. I have full backups for the addons, no problem. For the Wordpress database, blog entries dating back 15 years, I’m nervous that (*if* they’re restored) I shouldn’t also “backup” a blind copy to my “secret” Wordpress.com site. Else I can’t imagine how to maintain an off-host backup.

 

Ah, well, thanks so much for keeping us up to date pards. I’m sure you know it’s painful for us and I know it’s painful as well for you. I’m sorry my email of sympathy and gratitude generated a ticket. Unintended. Live long and prosper.

 

Hi, I find the program backup buddy by ithemes to be really helpful (no affiliation). I send my backups to my dropbox. Hope that helps.

Link to comment
Share on other sites

Please understand, no gripe, just tinfoil hat, okay? 5:12PM MDT and my Wordpress blog (plus two addon sites) are still down. I have full backups for the addons, no problem. For the Wordpress database, blog entries dating back 15 years, I’m nervous that (*if* they’re restored) I shouldn’t also “backup” a blind copy to my “secret” Wordpress.com site. Else I can’t imagine how to maintain an off-host backup.

 

Ah, well, thanks so much for keeping us up to date pards. I’m sure you know it’s painful for us and I know it’s painful as well for you. I’m sorry my email of sympathy and gratitude generated a ticket. Unintended. Live long and prosper.

 

Hi, I find the program backup buddy by ithemes to be really helpful (no affiliation). I send my backups to my dropbox. Hope that helps.

Link to comment
Share on other sites

Do you have plans to compensate customers for "pain and suffering"? Waiting up to 96 hours for a website to come back online is a little much to ask. As a developer, I have my customers calling me and asking about this. We could switch hosts, but wondering if you would give a refund, or how you plan to handle that situation.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...