Jump to content
MDDHosting Forums

Major Outage - 09/21/18+ - Client Discussion


KevinD872

Recommended Posts

I do NOT want to see "Service is down and will be fixed." I DO want to see a company being transparent and providing information along the way, including ETAs, if and when they are feasible, and explanations of changes in ETAs. In short, MDD provided all the information I wanted, perhaps more than I needed--but more is always better than less for me--and much much more than I could have expected in the circumstances. This was all deeply reassuring.

 

I have been with hosting providers that did the "It's down, it will be fixed" route, and *never again* is all I can say, because the corollary, though not expressed in precisely these terms, is generally "Now shut up and leave us alone, that's all we're going to tell you, and you're just going to have to wait it out without having a clue what's going on or how long it will take. Sucks to be you." I'm stuck with that kind of service with the cable company because their only competitor is just as bad, if not worse. Fortunately, MDD doesn't take that route, and I would never stay with a hosting provider that did

 

Different strokes for different folks.

  • Upvote 5
Link to comment
Share on other sites

I do NOT want to see "Service is down and will be fixed." I DO want to see a company being transparent and providing information along the way, including ETAs, if and when they are feasible, and explanations of changes in ETAs. In short, MDD provided all the information I wanted, perhaps more than I needed--but more is always better than less for me--and much much more than I could have expected in the circumstances. This was all deeply reassuring.

 

I have been with hosting providers that did the "It's down, it will be fixed" route, and *never again* is all I can say, because the corollary, though not expressed in precisely these terms, is generally "Now shut up and leave us alone, that's all we're going to tell you, and you're just going to have to wait it out without having a clue what's going on or how long it will take. Sucks to be you." I'm stuck with that kind of service with the cable company because their only competitor is just as bad, if not worse. Fortunately, MDD doesn't take that route, and I would never stay with a hosting provider that did

 

Different strokes for different folks.

 

 

i 100% agree with this....

 

MDD shows sincerity this is the business ill stay with

Link to comment
Share on other sites

I do NOT want to see "Service is down and will be fixed." I DO want to see a company being transparent and providing information along the way, including ETAs, if and when they are feasible, and explanations of changes in ETAs. In short, MDD provided all the information I wanted, perhaps more than I needed--but more is always better than less for me--and much much more than I could have expected in the circumstances. This was all deeply reassuring.

 

I have been with hosting providers that did the "It's down, it will be fixed" route, and *never again* is all I can say, because the corollary, though not expressed in precisely these terms, is generally "Now shut up and leave us alone, that's all we're going to tell you, and you're just going to have to wait it out without having a clue what's going on or how long it will take. Sucks to be you." I'm stuck with that kind of service with the cable company because their only competitor is just as bad, if not worse. Fortunately, MDD doesn't take that route, and I would never stay with a hosting provider that did

 

Different strokes for different folks.

 

Or blow smoke up your ***!

Also, 100% agree.

Link to comment
Share on other sites

I'd like a status update on S5. (Last one with %'s was four hours ago.)

 

I am grateful that Mike has been transparent and admitted to the errors and issues, and that he has identified ways to avoid this happening again. I do, however, have a lingering concern about how employees and processes are being managed and will be managed. That Mike didn't realize that snapshots weren't configured properly suggests to me that maybe he trusted someone to do this right. That numerous issues were encountered in the disaster-recovery process suggests to me that maybe it wasn't tested rigorously enough. With so much riding on their actions, IMO there should be a complete rethinking of this.

 

Think of a pilot and co-pilot in a commercial jet - as I understand it, they are both highly competent and capable of going through the pre-flight checklist on their own, but they don't - they do it together. Yes, they will spell each other occasionally during flight so one can go to the bathroom or deal with an issue in the cabin, but for the most part they are expected to be alert to the potential mistakes of the other, and there in real time to work on any issues that arise together. In addition, in designing plane systems, engineers have to ask, "What mistakes might pilots make", and they do their best to design systems and processes to not allow them make those mistakes.

 

I would be interested in hearing from Mike once this is all over what will change in terms of his management of people and processes.

Link to comment
Share on other sites

Restores are 100% Completed

 

If your site is offline showing a cPanel error page:

 

  • Try connecting to your cPanel by adding "/cpanel" on to the end of your domain. If you can sign in, this verifies your account was restored.
  • Check to see if you're using our nameservers - if you aren't, you'll need to get your IP from cPanel and update your third party DNS.
  • Make sure you're not just reloading the error page - hitting reload while viewing the error just reloads the error page.

If you are not using third party DNS and your site doesn't appear but you can get into cPanel - try clearing your browser cache and restarting your browser. If that doesn't work try another browser. If it loads for you on one browser but not another - that's a caching issue and not a server or network issue.

 

We do expect there to be a lot of little issues that we have to resolve so if you have issues and can't sort them please reach out in a ticket.

 

We are doing our best to keep up with support tickets. I am sorry if it takes us longer to reply than normal but we are answering tickets in the order received and doing our best to fully resolve any issues and to offer good proper non-copy-and-pasted advice.

Link to comment
Share on other sites

 

Restores are 100% Completed

 

If your site is offline showing a cPanel error page:

 

  • Try connecting to your cPanel by adding "/cpanel" on to the end of your domain. If you can sign in, this verifies your account was restored.
  • Check to see if you're using our nameservers - if you aren't, you'll need to get your IP from cPanel and update your third party DNS.
  • Make sure you're not just reloading the error page - hitting reload while viewing the error just reloads the error page.

If you are not using third party DNS and your site doesn't appear but you can get into cPanel - try clearing your browser cache and restarting your browser. If that doesn't work try another browser. If it loads for you on one browser but not another - that's a caching issue and not a server or network issue.

 

We do expect there to be a lot of little issues that we have to resolve so if you have issues and can't sort them please reach out in a ticket.

 

We are doing our best to keep up with support tickets. I am sorry if it takes us longer to reply than normal but we are answering tickets in the order received and doing our best to fully resolve any issues and to offer good proper non-copy-and-pasted advice.

 

 

Thank you for the update.

 

Can I ask as I don't want to bother you with a ticket. Will it take a while for things to settle down ?

 

All posts made on my forum within 2 - 3 hours of coming back on-line have gone, and we are also getting some intermittent connections still with some posts not getting completed.

Link to comment
Share on other sites

I am on S4 and I assume it is restored.

 

My static sites seem to work. However I have 2 joomla sites that still show this error message:

"SORRY! If you are the owner of this website, please contact your hosting provider..."

 

Is there something I should do?

 

I have those 2 joomla sites on CLOUDFLARE. I have disabled the caching and took them into development mode.

Link to comment
Share on other sites

I am on S4 and I assume it is restored.

 

My static sites seem to work. However I have 2 joomla sites that still show this error message:

"SORRY! If you are the owner of this website, please contact your hosting provider..."

 

Is there something I should do?

 

I have those 2 joomla sites on CLOUDFLARE. I have disabled the caching and took them into development mode.

Update the IP at CloudFlare to the one listed in your cPanel. If you can't sort this, open a ticket please.

Link to comment
Share on other sites

 

Thank you for the update.

 

Can I ask as I don't want to bother you with a ticket. Will it take a while for things to settle down ?

 

All posts made on my forum within 2 - 3 hours of coming back on-line have gone, and we are also getting some intermittent connections still with some posts not getting completed.

Go ahead and open a ticket.

Link to comment
Share on other sites

I was astonished to read the following statement on the locked outage forum today: "If anybody has a backup of their own - we can use that to get you online immediately. " I didn't know that. I wish it had been said in the MDDHosting emails about the outage. I meticulously keep a backup.

I was in the same boat. I also then thought I needed to have a full cPanel backup, but I didn't, so I assumed I was out of luck. Only Tuesday afternoon did I realize that I could restore my Wordpress backups. The team knows from the tickets that they get that clients run the gamut from web admin experts to those who don't know much (like me). While I wouldn't expect them to hold my hand while I restore my backups, just this little bit of information would have saved me 2-3 days of being down.

Link to comment
Share on other sites

I was in the same boat. I also then thought I needed to have a full cPanel backup, but I didn't, so I assumed I was out of luck. Only Tuesday afternoon did I realize that I could restore my Wordpress backups. The team knows from the tickets that they get that clients run the gamut from web admin experts to those who don't know much (like me). While I wouldn't expect them to hold my hand while I restore my backups, just this little bit of information would have saved me 2-3 days of being down.

Indeed. I realized not long ago that it was an assumption on my part that anybody that had backups of their data would ask to be able to restore that data.

 

I did tweet it a couple of times and I'm pretty sure I put it in the status thread as well - but no going back now :(.

Link to comment
Share on other sites

All servers are online and all accounts are restored!

 

We reached out to our storage platform vendor after the incident and we have worked with them to take steps to prevent an issue like this from happening again. Changes have also been implemented that will allow us to recover from a catastrophic event such as the one we just experienced as quickly as within a few minutes with little to no impact on the services themselves.

 

We are going to be conducting a thorough review of the events leading up to this incident and making changes to our policies and procedures based upon our findings. How the incident was handled is also going to be reviewed and we are going to develop a new comprehensive backup and emergency response plan.

 

If you are still experiencing any issues at all or need help with anything please do not hesitate to reach out to us. We are here to help and will do our best to assist you in recovering from this incident in any way that we can.

 

Thank you,

Link to comment
Share on other sites

I have been doing an assessment of what email we lost in the outage (our website is relatively static, so email is the much bigger issue for us).

I asked for and/or confirmed some information with the MDD staff. (Thanks @MikeDVB for the quick and personalized response). I thought the info might be helpful to others, but decided to make a separate topic so it could be found more easily.

 

https://forums.mddhosting.com/topic/1586-assessing-email-loss/

Link to comment
Share on other sites

Hi all

Quick question

Considering what happened it would be wise to have full cpanel backups.

Any suggestions on how to automate this process and maybe upload to amazon S3 bucket on a shared hosting account?

thanks!

We’re evaluating what options there are so that hopefully we can offer such functionality for you. I know it’s doable with a custom script of some kind but it would be nice for it to be built in.
  • Upvote 2
Link to comment
Share on other sites

I'm one of the people that had their own backup. Once I was made aware that there was an option to have a new account set up to get online again instantly I used that. It worked perfect.

 

However, two things:

 

- I would have liked to have been made aware of this option right away / sooner. I lost 4 days of business.

- This morning I received a message the server my account used to be hosted on (S3) was restored. Lucky for me I decided to see if my website was still online.

 

It wasn't. What happened was that my index page was overwritten by a general MDD dummy page. I have now replaced that page with my own page so it's all working again. But I lost another night of business (I'm in Europe but the site is aimed at the US).

 

Other people might have experienced the same thing - check if you haven't done so already !

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...