Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#1 KevinD872

KevinD872

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:NW Indiana
  • Interests:Drums & music, motorcycles, computers, programming and gadgets.

Posted 22 September 2018 - 11:57 AM

I can't speak for everyone, but while I'm not too happy about this, I can understand and empathize with your side of this situation.

 

That said, I do have one question:  How recent is the data backup on your "last resort disaster recovery" server(s)?


  • 0

#2 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,883 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 22 September 2018 - 11:57 AM

I can't speak for everyone, but while I'm not too happy about this, I can understand and empathize with your side of this situation.

 

That said, I do have one question:  How recent is the data backup on your "last resort disaster recovery" server(s)?

Most are only about 14 hours before the outage, one or two servers are 48 hours or so.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#3 KevinD872

KevinD872

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:NW Indiana
  • Interests:Drums & music, motorcycles, computers, programming and gadgets.

Posted 22 September 2018 - 12:01 PM

Most are only about 14 hours before the outage, one or two servers are 48 hours or so.

 

One last question:  What will happen with any emails we should have received between the problem occurring and when our sites come back online?


  • 0

#4 cziv

cziv

    Member

  • Members
  • PipPip
  • 52 posts
  • Gender:Male

Posted 22 September 2018 - 12:09 PM

Well mistakes happen but a recovery plan should have covered that. Quickly.

 

Snapshots were disabled, some backups are 14 hours some are 2 days.

 

Not a good plan guys. Not a good plan at all. In 2018.


  • 0

#5 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,883 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 22 September 2018 - 12:13 PM

 

One last question:  What will happen with any emails we should have received between the problem occurring and when our sites come back online?

 

It's up to the sending server to queue them and attempt delivery until the mail server comes back online.  I wish there were something we could do about it beyond that.

 

Well mistakes happen but a recovery plan should have covered that. Quickly.

 

Snapshots were disabled, some backups are 14 hours some are 2 days.

 

Not a good plan guys. Not a good plan at all. In 2018.

 

It's not that we had one thing go wrong - we had one thing, and then another, and then another, and then another.  We'll explain it all in the official RFO - but needless to say there should have been no situation where we actually had to restore from this particular disaster recovery system.
 


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#6 AMC4x4

AMC4x4

    Newbie

  • Members
  • Pip
  • 24 posts

Posted 22 September 2018 - 12:17 PM

It's not that we had one thing go wrong - we had one thing, and then another, and then another, and then another.  We'll explain it all in the official RFO - but needless to say there should have been no situation where we actually had to restore from this particular disaster recovery system.

 

 

I'm just grateful that your last resort backup actually had valid data and could be used. 96 hours I can deal with, but loss of data would be catastrophic for many of us. I'm sure we've all learned some things. For instance, I kept my backups on my MDD account. That was stupid on my part, but I thought your triple backups would be enough. This has been a huge wake up call for me to implement a regular personal backup routine that I should have been doing all along. I got complacent. 

Thanks again for all your hard work getting us back up and running, for the frequent updates, and for the transparency. 

-- still a customer for the long term


  • 0

#7 Devem

Devem

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 22 September 2018 - 12:20 PM

I guess MDD clients are mostly a patient lot. But if you are not making much or any money from your sites, maybe it's "ok".

But the fact is, just one day of downtime will cost (some) clients a lot of money, many times more than the credit refunds that MDD would ever give back as "compensation".

 

The worse case scenario (hope I'm wrong) is 3+ days of outage (it's extremely bad if you are earning from your sites).

 

I suspect when Michael said he may not get sleep for 72 hours, this is what he had in mind. :(


  • 0

#8 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,883 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 22 September 2018 - 12:28 PM

I guess MDD clients are mostly a patient lot. But if you are not making much or any money from your sites, maybe it's "ok".

But the fact is, just one day of downtime will cost (some) clients a lot of money, many times more than the credit refunds that MDD would ever give back as "compensation".

 

The worse case scenario (hope I'm wrong) is 3+ days of outage (it's extremely bad if you are earning from your sites).

 

I suspect when Michael said he may not get sleep for 72 hours, this is what he had in mind. :(

I hope for the best but plan for the worst.

 

We tried numerous things to speed this process up, which I will elaborate later as I'm short on time right now, but none of them worked.

 

There was one or two we could have tried that would have put the last set of data at risk - and we decided that it was better to have slower restorations than to possibly lose the data.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#9 AMC4x4

AMC4x4

    Newbie

  • Members
  • Pip
  • 24 posts

Posted 22 September 2018 - 12:29 PM

I guess MDD clients are mostly a patient lot. But if you are not making much or any money from your sites, maybe it's "ok".

But the fact is, just one day of downtime will cost (some) clients a lot of money, many times more than the credit refunds that MDD would ever give back as "compensation".

 

The worse case scenario (hope I'm wrong) is 3+ days of outage (it's extremely bad if you are earning from your sites).

 

I suspect when Michael said he may not get sleep for 72 hours, this is what he had in mind. :(

 

I bet Michael would differ with you on "patient lot," as it seems he's been getting lots of nasty messages. People are rightfully frustrated. Thing is, this company is way more transparent than any other company I've ever dealt with, and if there is any forgiving or patience on the part of customers, I'd say that's the only reason. We do tolerate a bit of downtime for the decent prices and excellent service, and it's because we know the MDD team is solid. They're the types of people we'd hire if we were in charge of hiring. Mistakes happen, and I'm sure they will learn from this.


  • 0

#10 Victoria Bampton

Victoria Bampton

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 22 September 2018 - 01:22 PM

@AMC4x4 well said. If it wasn't for the transparency, I'd be looking for a new host, but I'm not.

Human beings make mistakes, and it's how they respond to those mistakes that matters. It's absolutely clear that the MDD team are doing their level best in a really horrible situation. They're the kind of guys I want on my team.

For those complaints I've seen on Twitter - and I'm sure there's much worse on support tickets - all of us have had hair raising days in business, and if you haven't, you just haven't been in business long enough yet. Follow MDD's example, be transparent and honest. If your clients aren't willing to support you even though you're not perfect, they're not the clients you want.

@MikeDVB make sure your team get some sleep (and you too!). Exhausted people make bigger mistakes.
  • 1

#11 Rhody401

Rhody401

    Newbie

  • Members
  • Pip
  • 12 posts
  • Gender:Male
  • Location:Providence, RI USA
  • Interests:Consultant, Contractor, Beta Tester, IT Director.

Posted 22 September 2018 - 01:36 PM

Sometimes stuff happens.  We learn from it, and get better from the experience.

 

How a company responds in a crisis situation like this says the most, and you guys are doing a STELLAR job.  The transparency and dedication is greatly appreciated, and I won't nag or delay you by putting in a ticket. Everything I need to know was posted in these updates.

 

Thanks to the MDD staff, in advance, for another job well done.

 

Hopefully you'll get some well deserved sleep soon!

 

Mike


  • 1

#12 Laimonas

Laimonas

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male
  • Location:European Union
  • Interests:wine

Posted 22 September 2018 - 02:00 PM

Thanks for an update. Good luck in restoring data!

MDD site SAYS about daily offsite backups, I hope this was (and is) correct company's  statement.


  • 0

#13 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,883 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 22 September 2018 - 02:27 PM

Thanks for an update. Good luck in restoring data!

MDD site SAYS about daily offsite backups, I hope this was (and is) correct company's  statement.

That's what we're restoring from - our daily offsite backups.  If they didn't exist - we'd have zero data to restore.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#14 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 152 posts
  • Gender:Not Telling

Posted 22 September 2018 - 02:53 PM

Just want to encourage Mike and the team to keep up the good work.

 

I have been with MDD for 8 years and Mike always has and still is honest with his customers. Yes it's annoying and I too am getting a lot of flack form my forum members, but at the end of the day it's best to just let MDD get on with it.

 

I'm lucky as I don't  want to make money from my websites, but I do run a forum for people with a health problem and a lot of those people NEED my forum so it does effect us all in different ways.

 

No it shouldn't happen, but it has so now we just have to leave them to get on with it and get us all back on-line. Which I'm sure after my dealings with them that they will as soon as they can.

 

We can all learn from this and it's not till it goes wrong that we appreciate what we have. I still however feel safe with MDD and Mike should be proud of his honesty to his customers, I doubt many other owners would be that honest.

 

I still have full faith in Mike and his staff, I actually feel sorry for the person that pushed the wrong button so to speak. So knuckle down MDD and do your stuff, we are in your hands.


  • 0

#15 Laimonas

Laimonas

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male
  • Location:European Union
  • Interests:wine

Posted 22 September 2018 - 03:18 PM

I would prefer to see something different than IP address has changed, server misconfiguration or the site may have moved when trying to reach my site. Why not simply telling hosting provider is experiencing difficulties. Now all my clients think I caused a downtime myself and to be honest I am tired of phone calls my site is down and what am I going to do to make it alive, etc. 

Just my 2p.


  • 0

#16 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,883 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 22 September 2018 - 03:22 PM

I would prefer to see something different than IP address has changed, server misconfiguration or the site may have moved when trying to reach my site. Why not simply telling hosting provider is experiencing difficulties. Now all my clients think I caused a downtime myself and to be honest I am tired of phone calls my site is down and what am I going to do to make it alive, etc. 

Just my 2p.

We have a fair bit of clients ourselves and I understand fully about being tired of hearing about sites being down.  I mean it's not like you want them offline, right?  Neither do we.

 

I wish there was a faster way for us to get everything back online.

 

At the end of the day though it's our responsibility to support our customers.  I just wish we weren't getting 5 new tickets every 3 seconds - it's an untenable amount of incoming tickets to respond to.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#17 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,883 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 22 September 2018 - 03:42 PM

Our servers are exceptionally fast and can move data at gigabytes per second - it's our remote backup server that's slow and can't keep up.

 

I really honestly don't know how long this is going to take - that's the primary reason I keep not giving an ETA.

 

As it stands there are still thousands upon thousands of accounts to restore.  I'd like to say I'll get to sleep within the next few days but honesty I really don't know.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#18 Vilandra

Vilandra

    Newbie

  • Members
  • Pip
  • 23 posts
  • Gender:Female
  • Location:Pittsburgh, PA
  • Interests:Chelsea FC!

Posted 22 September 2018 - 03:45 PM

Thanks to the team for everything they are doing :)
  • 0

#19 AMC4x4

AMC4x4

    Newbie

  • Members
  • Pip
  • 24 posts

Posted 22 September 2018 - 03:53 PM

I would prefer to see something different than IP address has changed, server misconfiguration or the site may have moved when trying to reach my site. Why not simply telling hosting provider is experiencing difficulties. Now all my clients think I caused a downtime myself and to be honest I am tired of phone calls my site is down and what am I going to do to make it alive, etc. 

Just my 2p.

 

One of my sites was restored, but the SSL certificate is showing as invalid and it's redirecting to the MDD forums (if people click to accept the improper certificate), so perhaps clients will know that it's a hosting issue. Hopefully they don't think we've been hacked.


  • 0

#20 Tandemia

Tandemia

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 22 September 2018 - 04:37 PM

Can you please clarify the best/easiest way for us to check if an account has been restored without opening unnecessary support tickets?

Right now I am assuming that my account(s) has been not restored yet if I get an authentication error while trying to login to cPanel.
 

Thank you for being so open and transparent, much appreciated.


  • 0




6 user(s) are reading this topic

0 members, 6 guests, 0 anonymous users