Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#61 juliebolddogge

juliebolddogge

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Female

Posted 23 September 2018 - 12:56 PM

I agree with Newbie. My trust in MDD is stronger based on how they are responding to this crisis. I am a lifer as long as I have clients to host. I've struggled with how much to share with my clients and came up with this email. Feel free to use it if it will help you.  

 

- EMAIL THAT I SENT TO MY CLIENTS IF ANY OF YOU WOULD LIKE TO BORROW IT -

 

Dear All,

 

My hosting provider (where your sites are stored) is fighting an ongoing recovery process on a major outage at their server facility. First of all, your website data is fine - I have offsite backups so that is not an issue; however some of your sites are offline. Secondly, MDD is working on restoring the sites but they cannot state when (precisely) the sites will be back online, but say it should be within 24-48 hours. This is causing an inconvenience to each of you and for some, a loss of revenue, donations, and opportunities. For this, I am truly sorry and I am monitoring the situation closely.

 

I have trusted MDD Hosting for 8 years or more, and I can say that nothing like this has ever happened before. Their attention to customer service is unmatched in an industry where so many larger hosting companies care only about the margins. I care about the service and I believe that what has happened here is a very unfortunate sequence of events that has led to a disastrous outage. I remain diligent in following their progress in restoring your sites and will test each of them as they come back online. In the event that their backups are not as current as mine, I will restore my backups over theirs. I run backups daily in the event that something like this occurs so we are in good shape there.

 

The reason that I cannot simply restore the backup to the server now – instead of waiting for MDD – is because their restoration is at an account level. Until they restore the account and its respective cPanel, I have nothing to load the backup onto. I have asked them if I can simply create a new (temporary) account for my clients and restore my backups there but this is not possible. I’m doing everything that I can to see if I can provide some temporary relief. It is worth noting that MDD is improving their backup and restore process as a part of this experience.

 

One of the reasons that I trust MDD so much is that they are incredibly transparent in admitting mistakes and doing the right thing for the right reason. I have full faith in them now and going forward and I am 100% confident that this will not occur again. I say that based on all of the details that I’ve been following. If you are interested in the technical details, please let me know and I will gladly share them with you. 

 

I welcome your questions, comments, and concerns. Again, I am truly sorry about this situation. I value your business and appreciate your patience.

 

--> END

 

Thank you to all at MDD for your continued transparency and hard work! 


  • 0

#62 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 12:57 PM

The default page should be better now for when your site doesn't load should be better now.


  • 2
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#63 juliebolddogge

juliebolddogge

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Female

Posted 23 September 2018 - 01:01 PM

The default page should be better now for when your site doesn't load should be better now.

It is SOOOOO much better - thank you!!!!


  • 1

#64 Rhody401

Rhody401

    Newbie

  • Members
  • Pip
  • 12 posts
  • Gender:Male
  • Location:Providence, RI USA
  • Interests:Consultant, Contractor, Beta Tester, IT Director.

Posted 23 September 2018 - 01:02 PM

Thank you, Mike. That is MUCH better :)

 

Hopefully R1 isn't too far behind S1

techdiff.jpg


  • 2

#65 Rhody401

Rhody401

    Newbie

  • Members
  • Pip
  • 12 posts
  • Gender:Male
  • Location:Providence, RI USA
  • Interests:Consultant, Contractor, Beta Tester, IT Director.

Posted 23 September 2018 - 01:03 PM

dupe ok to zap


  • 1

#66 juliebolddogge

juliebolddogge

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Female

Posted 23 September 2018 - 01:05 PM

R1 - I just want to clarify when R1 will be restored. Is that after S1-S5? I ask because I think (?) my reseller accounts are all on R1, but I may be viewing this wrong. Take your time answering and thank you in advance. 


  • 0

#67 505MediaGroup

505MediaGroup

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 23 September 2018 - 01:13 PM

Hello,

This is my plea to fellow loyal customers. I've been with MDD hosting and Michael since 2012 and I don't plan on going anywhere. I've been hosting websites for 18 years and I've experienced the worst the industry has to offer.

 

MDD Hosting has been amazing and finding a company with the level of support is very hard. Michael is a pillar of business ethics and I trust what he says. All that being said.

 

Being in IT, I know how a catastrophic event in business can be soul crushing. A little help from enough of us loyal customers can ease the pain and ensure a good reputable business and small business owner with a young family and employees to support can survive it. Please take a look and consider helping out. https://www.gofundme...-in-crisis-help

 

If you can’t help Please Share. https://www.gofundme...-in-crisis-help

 

If you want to be on the FundRaising Team click here. 

https://www.gofundme...JKdwRg2cdS5qH7P

 

Please let me know if you have any questions.

 

Russ


  • 1

#68 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 23 September 2018 - 01:20 PM

The default page should be better now for when your site doesn't load should be better now.


Thanks Mike. So much better than the message it had before. Granted, the original issue is still present in the new wording. The message should not imply that WE as site owners are having a problem. Rather it should impress upon the visitor that our host provider is having problems.

 

Presently:
 

We are sorry but we are experiencing technical difficulties. We will be back online as soon as possible.

 

Should Be:

"We are sorry but our host provider is experiencing technical difficulties. We will be back online as soon as possible."


  • 0

#69 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 23 September 2018 - 01:29 PM

Hello,

This is my plea to fellow loyal customers. I've been with MDD hosting and Michael since 2012 and I don't plan on going anywhere. I've been hosting websites for 18 years and I've experienced the worst the industry has to offer.

 

MDD Hosting has been amazing and finding a company with the level of support is very hard. Michael is a pillar of business ethics and I trust what he says. All that being said.

 

Being in IT, I know how a catastrophic event in business can be soul crushing. A little help from enough of us loyal customers can ease the pain and ensure a good reputable business and small business owner with a young family and employees to support can survive it. Please take a look and consider helping out.

 

If you can’t help Please Share.

If you want to be on the FundRaising Team click here. 

 

 

Please let me know if you have any questions.

 

Russ

 

Unbelievable.  While we sit in disarray... 


  • 2

#70 ClassicalMark

ClassicalMark

    Newbie

  • Members
  • Pip
  • 6 posts
  • Gender:Male
  • Location:Denver CO

Posted 23 September 2018 - 01:31 PM

I and many of my clients are on s1.

 

The cPanel error page has gone, but I still can't get to the sites. I now get a different error page saying my browser can't establish a secure connection. I tried http: instead of https: but it made no difference.

 

When I try to open cPanel, I'm told my login is invalid.

 

I'm hoping these are just symptoms of the recovery process, and that once the sites are fully recovered I'll have access again.

 

Of course I'm impatient like everyone else, but like many of the members above, I've been nothing but delighted with MDD over the years and have no intention of changing hosts.

 

Mark


  • 0

_____________________________________________________________
iMac OS  High Sierra 10.13.6


#71 giantjack

giantjack

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 23 September 2018 - 02:02 PM

I have been trying to think of some way that we could automate this.  For each account create we have to kill the DNS Zone of the primary domain on the account and then we have to create the account.  Then we have to notify the client that we've done this as well as the new account details.  Beyond that we'll have to overwrite the accounts with our backups without overwriting anybody that's already been restored.  Additionally we'll have to be sure not to overwrite any accounts where the account was set up blank but the client decided to restore data and recover on their own.

 

If I can come up with a good way to do it that won't divert resources away from the restoration process we'll do it.  I can't think of any way to make it happen that doesn't involve communicating with us in some way, shape, or form.

 

We're presently waiting for the facility to move the disks from the old backup server to the new one.

 

If you go this path please be cautious when it comes to your resellers. We won't want a error message that says "hosting failure" or any reference to MDDhosting.com. I like the new error message page you've put up. Much more user friendly


  • 0

#72 giantjack

giantjack

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 23 September 2018 - 02:05 PM

We are working on getting the first server backups ready to restore directly to the server and are preparing to start copying the second server.  The second server is going to be P1, the third server is going to be R1.

 

So you're still on the 1st server? If it's going to take this long per server it's going to be days until all accounts are restored?

 

Also you say R1 will be 3rd but I have 2 domains on that account that have been restored. Would this suggest they were luckily restored manually?


  • 0

#73 Laimonas

Laimonas

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male
  • Location:European Union
  • Interests:wine

Posted 23 September 2018 - 02:08 PM

For me now looks much better! Thank you!


  • 0

#74 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 02:12 PM

 

If you go this path please be cautious when it comes to your resellers. We won't want a error message that says "hosting failure" or any reference to MDDhosting.com. I like the new error message page you've put up. Much more user friendly

 

We won't do it unless we can find a good way to do it that won't cause problems.  We do our best to keep everything white-label even our shared hosting.

 

 

So you're still on the 1st server? If it's going to take this long per server it's going to be days until all accounts are restored?

 

Also you say R1 will be 3rd but I have 2 domains on that account that have been restored. Would this suggest they were luckily restored manually?

 

We're copying the 2nd server to the SSD array and are preparing to restore the first server now.  It will take some time to copy each server's backups, as it's the old backup server that's the problem, but once copied the restoration process for each server should be very fast - on the order of magnitudes faster than simply restoring from the old backup server.

 

For me now looks much better! Thank you!

No problem.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#75 brucejordan

brucejordan

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 23 September 2018 - 02:44 PM

MDD, do you have any idea when R3 reseller servers will be restored.  I see different orders that you are posting but not this one.  Some of my websites are backup on this server but many are not.  DOes that mean you are still loading those?


  • 0

#76 webeye

webeye

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 23 September 2018 - 02:52 PM

I understand the pecking order at this time is S1, followed by P1, and then R1. How many servers are there? I'm on R3 - would it be last? Is the recovery time objective still up to 96 hours - 4 days?


  • 0

#77 jwinter

jwinter

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 23 September 2018 - 02:54 PM

my sites on P1 are running and have been since yesterday (Saturday) mid-afternoon.  Also sites on P2 are running, they were up before P1 sites were back.

 

are you going to restore over what is already there?   I have confirmed emails are working currently.  would another restore wipe out emails received at the server since it went back online?


  • 0

#78 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 02:57 PM

MDD, do you have any idea when R3 reseller servers will be restored.  I see different orders that you are posting but not this one.  Some of my websites are backup on this server but many are not.  DOes that mean you are still loading those?

We are restoring what we can as we can - this won't be an 'everybody is back online at the same time' type of thing - although I wish that were possible.

 

I understand the pecking order at this time is S1, followed by P1, and then R1. How many servers are there? I'm on R3 - would it be last? Is the recovery time objective still up to 96 hours - 4 days?

The current plan is S1, P1, R1.  After that we are likely going to go S2, P2, R2.  It is worth noting that S2, 3, 4, and 5 as well as P2, R2, R3, and R4 all hold far less data than the *1 servers - meaning each of those should go faster.

 

We're really hoping for no more than 48 hours - ideally 24 hours for most accounts to be online but we won't be able to provide a real ETA until we begin restoring backups from the new backup server to the client servers.  That's the only thing that's really potentially variable at this point.

 

mine sites on P1 are running and have been since yesterday (Saturday) mid-afternoon.  Also sites on P2 are running, they were up before P1 sites were back.

 

are you going to restore over what is already there?   I have confirmed emails are working currently.  would another restore wipe out emails received at the server since it went back online?

We won't be restoring over any already-restored accounts.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#79 giantjack

giantjack

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 23 September 2018 - 03:48 PM

Mike can you please confirm why some websites are redirecting to others on your network. E.g:

 

https://whmcsportal.com/

 

Once the SSL is accepted it'll redirect to another site unrelated to me


  • 0

#80 Tandemia

Tandemia

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 23 September 2018 - 03:52 PM

@MikeDVB Could you please clarify the status of the restoration of P1.

 

I am confused about jwinter reporting that his websites on P1 are okay while you saying that P1 backup image is being prepared for copy.

 

Thank you for your time.


  • 0




5 user(s) are reading this topic

0 members, 5 guests, 0 anonymous users