Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#381 LShoe

LShoe

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 26 September 2018 - 12:46 PM

I'd like a status update on S5. (Last one with %'s was four hours ago.)

 

I am grateful that Mike has been transparent and admitted to the errors and issues, and that he has identified ways to avoid this happening again. I do, however, have a lingering concern about how employees and processes are being managed and will be managed. That Mike didn't realize that snapshots weren't configured properly suggests to me that maybe he trusted someone to do this right. That numerous issues were encountered in the disaster-recovery process suggests to me that maybe it wasn't tested rigorously enough. With so much riding on  their actions, IMO there should be a complete rethinking of this. 

 

Think of a pilot and co-pilot in a commercial jet  - as I understand it, they are both highly competent and capable of going through the pre-flight checklist on their own, but they don't - they do it together. Yes, they will spell each other occasionally during flight so one can go to the bathroom or deal with an issue in the cabin, but for the most part they are expected to be alert to the potential mistakes of the other, and there in real time to work on any issues that arise together. In addition, in designing plane systems, engineers have to ask, "What mistakes might pilots make", and they do their best to design systems and processes to not allow them make those mistakes. 

 

I would be interested in hearing from Mike once this is all over what will change in terms of his management of people and processes.


  • 0

#382 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 September 2018 - 01:35 PM

Restores are 100% Completed

 

If your site is offline showing a cPanel error page:

 

  • Try connecting to your cPanel by adding "/cpanel" on to the end of your domain.  If you can sign in, this verifies your account was restored.
  • Check to see if you're using our nameservers - if you aren't, you'll need to get your IP from cPanel and update your third party DNS.
  • Make sure you're not just reloading the error page - hitting reload while viewing the error just reloads the error page.

If you are not using third party DNS and your site doesn't appear but you can get into cPanel - try clearing your browser cache and restarting your browser.  If that doesn't work try another browser.  If it loads for you on one browser but not another - that's a caching issue and not a server or network issue.

 

We do expect there to be a lot of little issues that we have to resolve so if you have issues and can't sort them please reach out in a ticket.

 

We are doing our best to keep up with support tickets.  I am sorry if it takes us longer to reply than normal but we are answering tickets in the order received and doing our best to fully resolve any issues and to offer good proper non-copy-and-pasted advice.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#383 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 156 posts
  • Gender:Not Telling

Posted 26 September 2018 - 01:56 PM

 

Restores are 100% Completed

 

If your site is offline showing a cPanel error page:

 

  • Try connecting to your cPanel by adding "/cpanel" on to the end of your domain.  If you can sign in, this verifies your account was restored.
  • Check to see if you're using our nameservers - if you aren't, you'll need to get your IP from cPanel and update your third party DNS.
  • Make sure you're not just reloading the error page - hitting reload while viewing the error just reloads the error page.

If you are not using third party DNS and your site doesn't appear but you can get into cPanel - try clearing your browser cache and restarting your browser.  If that doesn't work try another browser.  If it loads for you on one browser but not another - that's a caching issue and not a server or network issue.

 

We do expect there to be a lot of little issues that we have to resolve so if you have issues and can't sort them please reach out in a ticket.

 

We are doing our best to keep up with support tickets.  I am sorry if it takes us longer to reply than normal but we are answering tickets in the order received and doing our best to fully resolve any issues and to offer good proper non-copy-and-pasted advice.

 

 

Thank you for the update.

 

Can I ask as I don't want to bother you with  a ticket. Will it take a while for things to settle down ?

 

All posts made on my forum within 2 - 3 hours of coming back on-line have gone, and we are also getting some intermittent connections still with some posts not getting completed.


  • 0

#384 bobptz

bobptz

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 26 September 2018 - 02:00 PM

I am on S4 and I assume it is restored.

 

My static sites seem to work.  However I have 2 joomla sites that still show this error message:

"SORRY! If you are the owner of this website, please contact your hosting provider..."

 

Is there something I should do?

 

I have those 2 joomla sites on CLOUDFLARE.  I have disabled the caching and took them into development mode.


  • 0

#385 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 September 2018 - 02:09 PM

I am on S4 and I assume it is restored.

 

My static sites seem to work.  However I have 2 joomla sites that still show this error message:

"SORRY! If you are the owner of this website, please contact your hosting provider..."

 

Is there something I should do?

 

I have those 2 joomla sites on CLOUDFLARE.  I have disabled the caching and took them into development mode.

Update the IP at CloudFlare to the one listed in your cPanel.  If you can't sort this, open a ticket please.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#386 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 September 2018 - 02:10 PM

 

Thank you for the update.

 

Can I ask as I don't want to bother you with  a ticket. Will it take a while for things to settle down ?

 

All posts made on my forum within 2 - 3 hours of coming back on-line have gone, and we are also getting some intermittent connections still with some posts not getting completed.

Go ahead and open a ticket.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#387 sf2099

sf2099

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 26 September 2018 - 02:13 PM

Is anyone on the S2 server experiencing sending emails from a desktop client (i.e. Outlook or Thunderbird)?

 

It was working for me until about 45 minutes ago.  just wondering if its me or not...


  • 0

#388 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 156 posts
  • Gender:Not Telling

Posted 26 September 2018 - 02:29 PM

Go ahead and open a ticket.

 

Ticket Created #982690


  • 0

#389 bobptz

bobptz

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 26 September 2018 - 02:57 PM

Update the IP at CloudFlare to the one listed in your cPanel.  If you can't sort this, open a ticket please.

Worked, than you.


  • 0

#390 LShoe

LShoe

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 26 September 2018 - 06:45 PM

I was astonished to read the following statement on the locked outage forum today:  "If anybody has a backup of their own - we can use that to get you online immediately. "  I didn't know that. I wish it had been said in the MDDHosting emails about the outage. I meticulously keep a backup.  

I was in the same boat. I also then thought I needed to have a full cPanel backup, but I didn't, so I assumed I was out of luck. Only Tuesday afternoon did I realize that I could restore my Wordpress backups.  The team knows from the tickets that they get that clients run the gamut from web admin experts to those who don't know much (like me). While I wouldn't expect them to hold my hand while I restore my backups, just this little bit of information would have saved me 2-3 days of being down. 


  • 0

#391 LShoe

LShoe

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 26 September 2018 - 06:46 PM

Speaking of full cPanel backup, which option is this in cPanel?


  • 0

#392 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 September 2018 - 06:46 PM

I was in the same boat. I also then thought I needed to have a full cPanel backup, but I didn't, so I assumed I was out of luck. Only Tuesday afternoon did I realize that I could restore my Wordpress backups.  The team knows from the tickets that they get that clients run the gamut from web admin experts to those who don't know much (like me). While I wouldn't expect them to hold my hand while I restore my backups, just this little bit of information would have saved me 2-3 days of being down. 

Indeed.  I realized not long ago that it was an assumption on my part that anybody that had backups of their data would ask to be able to restore that data.

 

I did tweet it a couple of times and I'm pretty sure I put it in the status thread as well - but no going back now :(.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#393 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 September 2018 - 06:47 PM

Speaking of full cPanel backup, which option is this in cPanel?

cPanel -> Files -> Backup -> Download a Full Website Backup.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#394 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 September 2018 - 09:29 PM

All servers are online and all accounts are restored!

We reached out to our storage platform vendor after the incident and we have worked with them to take steps to prevent an issue like this from happening again. Changes have also been implemented that will allow us to recover from a catastrophic event such as the one we just experienced as quickly as within a few minutes with little to no impact on the services themselves.

We are going to be conducting a thorough review of the events leading up to this incident and making changes to our policies and procedures based upon our findings.  How the incident was handled is also going to be reviewed and we are going to develop a new comprehensive backup and emergency response plan.

If you are still experiencing any issues at all or need help with anything please do not hesitate to reach out to us.  We are here to help and will do our best to assist you in recovering from this incident in any way that we can.

Thank you,


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#395 cweinhofer

cweinhofer

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 26 September 2018 - 09:36 PM

I have been doing an assessment of what email we lost in the outage (our website is relatively static, so email is the much bigger issue for us).

I asked for and/or confirmed some information with the MDD staff. (Thanks @MikeDVB for the quick and personalized response). I thought the info might be helpful to others, but decided to make a separate topic so it could be found more easily.

 

https://forums.mddho...ing-email-loss/


  • 0

#396 cweinhofer

cweinhofer

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 26 September 2018 - 09:56 PM

Sorry, looks like there was some problem with my original topic. Here is the working link: https://forums.mddho...ing-email-loss/


  • 0

#397 teacdan

teacdan

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 26 September 2018 - 11:55 PM

Hi all

Quick question

Considering what happened it would be wise to have full cpanel backups.

Any suggestions on how to automate this process and maybe upload to amazon S3 bucket on a shared hosting account?

thanks!


  • 0

#398 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 27 September 2018 - 01:07 AM

Hi all
Quick question
Considering what happened it would be wise to have full cpanel backups.
Any suggestions on how to automate this process and maybe upload to amazon S3 bucket on a shared hosting account?
thanks!

We’re evaluating what options there are so that hopefully we can offer such functionality for you. I know it’s doable with a custom script of some kind but it would be nice for it to be built in.
  • 2
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#399 Maal

Maal

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 27 September 2018 - 03:20 AM

I'm one of the people that had their own backup. Once I was made aware that there was an option to have a new account set up to get online again instantly I used that. It worked perfect.

 

However, two things:

 

-  I would have liked to have been made aware of this option right away / sooner. I lost 4 days of business.

- This morning I received a message the server my account used to be hosted on (S3) was restored. Lucky for me I decided to see if my website was still online.

 

It wasn't. What happened was that my index page was overwritten by a general MDD dummy page. I have now replaced that page with my own page so it's all working again. But I lost another night of business (I'm in Europe but the site is aimed at the US).

 

Other people might have experienced the same thing - check if you haven't done so already !


  • 0

#400 Laimonas

Laimonas

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male
  • Location:European Union
  • Interests:wine

Posted 27 September 2018 - 04:51 AM

I am noob in servers, but would be interesting to know when performing a _block discard_.which caused this outage, doesn't system asks "do you really like to discard" or everything goes immediately?


  • 0




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users