Jump to content
MDDHosting Forums

Major Outage - 09/21/18+ - Client Discussion


KevinD872

Recommended Posts

I've been with MDD for a little over 10 years, and I can definitely vouch for their normally rock-solid service, and they definitely provide excellent support under normal circumstances. With this said, I have no intentions of leaving MDD and if anything I think I will actually have even more confidence after the dust settles since I have no doubt that they will do whatever they can to prevent such a catastrophe in the future.

 

I'm also glad to hear that MDD still has no plans to sell the business. I remember specifically asking about this when I first signed up because it is true that this almost always results in the provider's quality of service taking a nosedive. I've been burned before, and I know this. I hope and trust that should your son(s) indeed take over one day that they will have as much integrity and pride in their service as you do.

 

Regarding the less than desirable message that appears when customers attempt to visit our websites:

Rather than having us clients create yet another ticket just so we can replace it with a better message or temporary website, would it be feasible to just do this for us all on our behalf? I'm sure I'm not alone in thinking that a message stating "The host provider for this website has suffered a catastrophic failure and we are working around the clock to restore service ASAP. We sincerely apologize for the inconvenience." would be better than it is now as it looks more like the website was abandoned or left misconfigured for days. I don't know what all would be involved in order to do this for all clients, but on the surface anyway, it would only be one single and small HTML "index" file to copy to each domain's public folder.

  • Upvote 1
Link to comment
Share on other sites

As it stands right now - I'm just watching a copy bar progress - there's nothing at this moment I can do to assist with or speed up the process of restoring services or I would be doing just that.

 

I will be posting status updates here on this very thread.

 

Regarding the less than desirable message that appears when customers attempt to visit our websites:

Rather than having us clients create yet another ticket just so we can replace it with a better message or temporary website, would it be feasible to just do this for us all on our behalf? I'm sure I'm not alone in thinking that a message stating "The host provider for this website has suffered a catastrophic failure and we are working around the clock to restore service ASAP. We sincerely apologize for the inconvenience." would be better than it is now as it looks more like the website was abandoned or left misconfigured for days. I don't know what all would be involved in order to do this for all clients, but on the surface anyway, it would only be one single and small HTML "index" file to copy to each domain's public folder.

Very much agree. It makes more sense to go ahead and implement this servers wide as a gesture of good will and to preserve our reputations.

 

In the meantime I've sent a ticket asking for copies so I can consider alternatives given that I am on server 5 and Server 1 is only just completed.

Link to comment
Share on other sites

I've been with MDD for a little over 10 years, and I can definitely vouch for their normally rock-solid service, and they definitely provide excellent support under normal circumstances. With this said, I have no intentions of leaving MDD and if anything I think I will actually have even more confidence after the dust settles since I have no doubt that they will do whatever they can to prevent such a catastrophe in the future.

 

I'm also glad to hear that MDD still has no plans to sell the business. I remember specifically asking about this when I first signed up because it is true that this almost always results in the provider's quality of service taking a nosedive. I've been burned before, and I know this. I hope and trust that should your son(s) indeed take over one day that they will have as much integrity and pride in their service as you do.

 

Regarding the less than desirable message that appears when customers attempt to visit our websites:

Rather than having us clients create yet another ticket just so we can replace it with a better message or temporary website, would it be feasible to just do this for us all on our behalf? I'm sure I'm not alone in thinking that a message stating "The host provider for this website has suffered a catastrophic failure and we are working around the clock to restore service ASAP. We sincerely apologize for the inconvenience." would be better than it is now as it looks more like the website was abandoned or left misconfigured for days. I don't know what all would be involved in order to do this for all clients, but on the surface anyway, it would only be one single and small HTML "index" file to copy to each domain's public folder.

 

 

 

Very much agree. It makes more sense to go ahead and implement this servers wide as a gesture of good will and to preserve our reputations.

 

In the meantime I've sent a ticket asking for copies so I can consider alternatives given that I am on server 5 and Server 1 is only just completed.

I have been trying to think of some way that we could automate this. For each account create we have to kill the DNS Zone of the primary domain on the account and then we have to create the account. Then we have to notify the client that we've done this as well as the new account details. Beyond that we'll have to overwrite the accounts with our backups without overwriting anybody that's already been restored. Additionally we'll have to be sure not to overwrite any accounts where the account was set up blank but the client decided to restore data and recover on their own.

 

If I can come up with a good way to do it that won't divert resources away from the restoration process we'll do it. I can't think of any way to make it happen that doesn't involve communicating with us in some way, shape, or form.

 

Is the restoration of S1 going well ?

We're presently waiting for the facility to move the disks from the old backup server to the new one.

Link to comment
Share on other sites

I have been trying to think of some way that we could automate this. For each account create we have to kill the DNS Zone of the primary domain on the account and then we have to create the account. Then we have to notify the client that we've done this as well as the new account details. Beyond that we'll have to overwrite the accounts with our backups without overwriting anybody that's already been restored. Additionally we'll have to be sure not to overwrite any accounts where the account was set up blank but the client decided to restore data and recover on their own.

 

If I can come up with a good way to do it that won't divert resources away from the restoration process we'll do it. I can't think of any way to make it happen that doesn't involve communicating with us in some way, shape, or form.

 

We're presently waiting for the facility to move the disks from the old backup server to the new one.

 

 

Can you edit the "cgi-sys/defaultwebpage.cgi" file? Since it is already there, there's no need to kill/create/overwrite anything.

Link to comment
Share on other sites

Michael,

 

Have you just implemented an automated message for tickets? It is now sounding spammy and my ticket has had 6 automated replies in a row.

 

I rather you do not implement this automated response.

 

We are sorry for not being able to respond to support tickets as quickly as we normally do. We are doing our best to answer everybody in detail.

All tickets are presently being answered in the order they are received.

Link to comment
Share on other sites

Already working on that.

Good to hear. It sounds like this is likely the best way to go. (Thanks for the idea, Brad!)

 

If you do this, just be sure to change it back to the original default once this is all said and done. Otherwise it will just be confusing when the real reason is indeed a configuration problem. :D

Link to comment
Share on other sites

I agree with Newbie. My trust in MDD is stronger based on how they are responding to this crisis. I am a lifer as long as I have clients to host. I've struggled with how much to share with my clients and came up with this email. Feel free to use it if it will help you.

 

- EMAIL THAT I SENT TO MY CLIENTS IF ANY OF YOU WOULD LIKE TO BORROW IT -

 

Dear All,

 

My hosting provider (where your sites are stored) is fighting an ongoing recovery process on a major outage at their server facility. First of all, your website data is fine - I have offsite backups so that is not an issue; however some of your sites are offline. Secondly, MDD is working on restoring the sites but they cannot state when (precisely) the sites will be back online, but say it should be within 24-48 hours. This is causing an inconvenience to each of you and for some, a loss of revenue, donations, and opportunities. For this, I am truly sorry and I am monitoring the situation closely.

 

I have trusted MDD Hosting for 8 years or more, and I can say that nothing like this has ever happened before. Their attention to customer service is unmatched in an industry where so many larger hosting companies care only about the margins. I care about the service and I believe that what has happened here is a very unfortunate sequence of events that has led to a disastrous outage. I remain diligent in following their progress in restoring your sites and will test each of them as they come back online. In the event that their backups are not as current as mine, I will restore my backups over theirs. I run backups daily in the event that something like this occurs so we are in good shape there.

 

The reason that I cannot simply restore the backup to the server now – instead of waiting for MDD – is because their restoration is at an account level. Until they restore the account and its respective cPanel, I have nothing to load the backup onto. I have asked them if I can simply create a new (temporary) account for my clients and restore my backups there but this is not possible. I’m doing everything that I can to see if I can provide some temporary relief. It is worth noting that MDD is improving their backup and restore process as a part of this experience.

 

One of the reasons that I trust MDD so much is that they are incredibly transparent in admitting mistakes and doing the right thing for the right reason. I have full faith in them now and going forward and I am 100% confident that this will not occur again. I say that based on all of the details that I’ve been following. If you are interested in the technical details, please let me know and I will gladly share them with you.

 

I welcome your questions, comments, and concerns. Again, I am truly sorry about this situation. I value your business and appreciate your patience.

 

--> END

 

Thank you to all at MDD for your continued transparency and hard work!

Link to comment
Share on other sites

Hello,

This is my plea to fellow loyal customers. I've been with MDD hosting and Michael since 2012 and I don't plan on going anywhere. I've been hosting websites for 18 years and I've experienced the worst the industry has to offer.

 

MDD Hosting has been amazing and finding a company with the level of support is very hard. Michael is a pillar of business ethics and I trust what he says. All that being said.

 

Being in IT, I know how a catastrophic event in business can be soul crushing. A little help from enough of us loyal customers can ease the pain and ensure a good reputable business and small business owner with a young family and employees to support can survive it. Please take a look and consider helping out. https://www.gofundme.com/fellow-tech-in-crisis-help

 

If you can’t help Please Share. https://www.gofundme.com/fellow-tech-in-crisis-help

 

If you want to be on the FundRaising Team click here.

https://www.gofundme.com/ukaeyh-help-a-fellow-techie-brother-out?teamInvite=36NWXIpg1hGUrZW8OY1Y9qmcGnL18iEyIdtPzNgNxJzmLO5WDJKdwRg2cdS5qH7P

 

Please let me know if you have any questions.

 

Russ

  • Upvote 1
Link to comment
Share on other sites

The default page should be better now for when your site doesn't load should be better now.

Thanks Mike. So much better than the message it had before. Granted, the original issue is still present in the new wording. The message should not imply that WE as site owners are having a problem. Rather it should impress upon the visitor that our host provider is having problems.

 

Presently:

 

We are sorry but we are experiencing technical difficulties. We will be back online as soon as possible.

 

Should Be:

"We are sorry but our host provider is experiencing technical difficulties. We will be back online as soon as possible."

Link to comment
Share on other sites

Hello,

This is my plea to fellow loyal customers. I've been with MDD hosting and Michael since 2012 and I don't plan on going anywhere. I've been hosting websites for 18 years and I've experienced the worst the industry has to offer.

 

MDD Hosting has been amazing and finding a company with the level of support is very hard. Michael is a pillar of business ethics and I trust what he says. All that being said.

 

Being in IT, I know how a catastrophic event in business can be soul crushing. A little help from enough of us loyal customers can ease the pain and ensure a good reputable business and small business owner with a young family and employees to support can survive it. Please take a look and consider helping out.

 

If you can’t help Please Share.

If you want to be on the FundRaising Team click here.

 

 

Please let me know if you have any questions.

 

Russ

 

Unbelievable. While we sit in disarray...

  • Upvote 2
Link to comment
Share on other sites

I and many of my clients are on s1.

 

The cPanel error page has gone, but I still can't get to the sites. I now get a different error page saying my browser can't establish a secure connection. I tried http: instead of https: but it made no difference.

 

When I try to open cPanel, I'm told my login is invalid.

 

I'm hoping these are just symptoms of the recovery process, and that once the sites are fully recovered I'll have access again.

 

Of course I'm impatient like everyone else, but like many of the members above, I've been nothing but delighted with MDD over the years and have no intention of changing hosts.

 

Mark

Link to comment
Share on other sites

I have been trying to think of some way that we could automate this. For each account create we have to kill the DNS Zone of the primary domain on the account and then we have to create the account. Then we have to notify the client that we've done this as well as the new account details. Beyond that we'll have to overwrite the accounts with our backups without overwriting anybody that's already been restored. Additionally we'll have to be sure not to overwrite any accounts where the account was set up blank but the client decided to restore data and recover on their own.

 

If I can come up with a good way to do it that won't divert resources away from the restoration process we'll do it. I can't think of any way to make it happen that doesn't involve communicating with us in some way, shape, or form.

 

We're presently waiting for the facility to move the disks from the old backup server to the new one.

 

If you go this path please be cautious when it comes to your resellers. We won't want a error message that says "hosting failure" or any reference to MDDhosting.com. I like the new error message page you've put up. Much more user friendly

Link to comment
Share on other sites

We are working on getting the first server backups ready to restore directly to the server and are preparing to start copying the second server. The second server is going to be P1, the third server is going to be R1.

 

So you're still on the 1st server? If it's going to take this long per server it's going to be days until all accounts are restored?

 

Also you say R1 will be 3rd but I have 2 domains on that account that have been restored. Would this suggest they were luckily restored manually?

Link to comment
Share on other sites

 

If you go this path please be cautious when it comes to your resellers. We won't want a error message that says "hosting failure" or any reference to MDDhosting.com. I like the new error message page you've put up. Much more user friendly

 

We won't do it unless we can find a good way to do it that won't cause problems. We do our best to keep everything white-label even our shared hosting.

 

 

So you're still on the 1st server? If it's going to take this long per server it's going to be days until all accounts are restored?

 

Also you say R1 will be 3rd but I have 2 domains on that account that have been restored. Would this suggest they were luckily restored manually?

 

We're copying the 2nd server to the SSD array and are preparing to restore the first server now. It will take some time to copy each server's backups, as it's the old backup server that's the problem, but once copied the restoration process for each server should be very fast - on the order of magnitudes faster than simply restoring from the old backup server.

 

For me now looks much better! Thank you!

No problem.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...