Major Outage - 09/21/18+ - Client Discussion

[Kevin]

Does anybody know the date of the "last chance/offsite" backup?

"Most are only about 14 hours before the outage, one or two servers are 48 hours or so." - https://forums.mddhosting.com/topic/1583-major-outage-092118-client-discussion/?p=6923

[CLINECO]

"Most are only about 14 hours before the outage, one or two servers are 48 hours or so." - https://forums.mddhosting.com/topic/1583-major-outage-092118-client-discussion/?p=6923

Thanks Kevin

[Michael D.]

Most are on September 20 - the night before the incident. Some are on the 21st early morning. A few on the 19th or 18th - and it varies from account to account but most are on the latest times.

 

The worst I've seen was 4 days in the past.

[MisterNeutron]

My site (ooakforum.com) is on P1, but it's still not working after almost 12h.

 

It took some fairly aggressive cache-clearing in Firefox, but your forum then came up just fine - latest post is from tomorrow (ah, the mysteries of the International Date Line!).

[Rhody401]

I'm starting to see some LIFE on one of my R1 accounts. I just received backed up (re-sent) emails from that domain from 9/17 until 2:30pm. (so I'm assuming Phoenix backup was from about then, and I'm one that is 4 days in the past) I can live with that

 

I can get into that account with FTP(and see all my files), but web browser brings me to the original cpanel "Sorry..." message. (instead of the modified one that was up earlier). I tried with a different browser and can get in, so probably just a cache.

 

Thank you!

[vask7]

Thanks for your commitment Michael.

 

Are we waiting for a full restore before we can access WHM?

 

Moreover, is there a way we can force redirect the default webpage back to / , and force clear the browser-cached redirection?

 

Thanks again for your efforts.

[bmolnar]

Thanks for your commitment Michael.

 

Are we waiting for a full restore before we can access WHM?

 

Moreover, is there a way we can force redirect the default webpage back to / , and force clear the browser-cached redirection?

 

Thanks again for your efforts.

I have no way to test this since my site on S3 is still down, but you should be able to put this in the .htaccess file in /www

<FilesMatch "\.(css|flv|gif|htm|html|ico|jpe|jpeg|jpg|js|png|pdf|swf|txt)$">
	<IfModule mod_expires.c>
		ExpiresActive Off
	</IfModule>
	<IfModule mod_headers.c>
		FileETag None
		Header unset ETag
		Header unset Pragma
		Header unset Cache-Control
		Header unset Last-Modified
		Header set Pragma "no-cache"
		Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate"
		Header set Expires "Mon, 10 Apr 1972 00:00:00 GMT"
	</IfModule>
</FilesMatch>

On a side-note, from any server that hasn't been restored yet, are any of you able to login to Cpanel or do we have to wait? Thanks

[spocklikesspores]

Did I miss something or was the last ETA/progress posting from MDD early this AM? By now there should be additional information...is it being posted somewhere that I'm not tuned in to?

[bmolnar]

Did I miss something or was the last ETA/progress posting from MDD early this AM? By now there should be additional information...is it being posted somewhere that I'm not tuned in to?

 

Updates from staff are being posted in another thread: https://forums.mddhosting.com/topic/1582-major-outage-092118-09242018/ This thread is just for discussion

[Michael D.]

Thanks for your commitment Michael.

 

Are we waiting for a full restore before we can access WHM?

 

Moreover, is there a way we can force redirect the default webpage back to / , and force clear the browser-cached redirection?

 

Thanks again for your efforts.

You should have access to WHM but if you don't - open a ticket and we can reset permissions.

 

Unfortunately the cgi-sys folder isn't actually loaded from your account so you can't do anything to affect it.

 

 

Updates from staff are being posted in another thread: https://forums.mddhosting.com/topic/1582-major-outage-092118-09242018/ This thread is just for discussion

Correct - we split them out because it was becoming impossible for people to keep up with the status updates we provide - but we didn't want to kill the conversation either.

[bmolnar]

Mike, is it safe for me to assume my Cpanel login on S3 won't work the server restore is complete? Just want to make sure it's not a bad password. Guess it's just luck of the draw being a client for 8 years but being on a newer server Oh well, just means it's less busy I suppose. Thanks!

[alpachinco]

Hi, I am on the P1 server which seems to be done, thank you, Mike and team, for all your hard work.

 

I have a few questions:

1. Do I need to change my email settings? The cPanel settings show incoming & outgoing servers listed as mail.mydomain, the previous settings were p1.supportedns.com. Which is correct?

2. Will the emails that people sent to me during the outage reach me eventually or are those completely gone?

3. My website is still reading a "404 File Not Found" error message. As the P1 server is up and running how do I get my site back online? Can specific instructions be sent, please?

 

Thank you MDD team.

[mdduser]

I would also like to know if I need to open a ticket for an invalid WHM password on S2. I just want to hit the ground running when my clients' accounts get back online. Thank you.

[Michael D.]

Yes, if you don't have WHM access you need a ticket. It's a quick fix when brought to our attention. We're trying to repair it proactively as well.

[Nitish]

Hi Michael,

If a website starts loading all the pages and I can login successfully into the cPanel, does this mean that it is alright to resume working on it or should I wait a couple of days.

Human errors happen every now and then. I appreciate the way MDD hosting handled this issue with full transparency.

Just a couple of requests. Once everything goes back to normal, could you please write a post on the forum or send an email about the steps that you are taking to

1. Make sure something like this does not happen again.
2. If it does happen, the data can be restored as quickly as possible?

Thanks.

[KevinD872]

Hi, I am on the P1 server which seems to be done, thank you, Mike and team, for all your hard work.

 

I have a few questions:

1. Do I need to change my email settings? The cPanel settings show incoming & outgoing servers listed as mail.mydomain, the previous settings were p1.supportedns.com. Which is correct?

2. Will the emails that people sent to me during the outage reach me eventually or are those completely gone?

3. My website is still reading a "404 File Not Found" error message. As the P1 server is up and running how do I get my site back online? Can specific instructions be sent, please?

 

Thank you MDD team.

 

Until you get a more proper response, I will throw my 2 cents in:

 

1 - I don't think it matters which server you use, as in the end, it is the same server. I have used both interchangeably and it works just the same. I would guess that perhaps the xx.supportedns.com may be the best option as using your domain name would just add one more link in the chain but I don't really know that for sure. Even so, the speed impact would be negligible.

 

2 - The sending server will keep trying to send any emails to you, but they all give up eventually. Either way, the sender would generally get a "bounce back" email saying that the message could not be delivered. After that, it would be up to them if they want to try sending it again. I think it's safe to say that for as long as this outage has lasted, everyone will be losing at least some messages, unless they had a personal backup that they had MDD apply to their account via a support ticket very early on before any sending servers would have completely given up on trying to send the email.

 

3 - You could try clearing your browser cache (or try an alternate browser such as IE, Chrome, Firefox, etc. to see if it goes away) or perhaps someone else may have a better answer.

[Michael D.]

Already planning on that. That said it shouldn't be possible to happen again but we'll detail how when we have more time to write the official document.

 

We're also making big change in our backup strategies to make sure restorations are fast if we ever have to resort to them again.

[NiKEUS]

Is there a reasonably priced backup provider of whm/cpanel that's automated etc that anyone has stumbled across?

[cziv]

 

I'm running a SEO agency for over 10 years, so I may share some details on this: It's pure luck how fast Google de-indexes a site if it's offline since it depends on when the site was last crawled and how fast Google processes the data to the live index. Can happen in a few hours, can take weeks till Google recognizes the site is gone. De-indexing is no penalty (although some penalties can cause de-indexing, but that's a diffent topic and doesn't matter here), Google simply doesn't want to show people an error page since that's a bad user experience.

 

The good news: Once the site is back online, it will return to/in the old positions. How fast depends on the same factors as de-indexing, the more popular the site, the better it's linked from external sites, the more often it gets crawled, the faster it will re-appear in the SERPs. Worst case downtime and a new IP may trigger a re-evaluation with a ranking rollercoaster ride for a few weeks till it settles in the old positions. Best case, the site instantly ranks like before. So, there is no permanent damage due to the downtime, but you may experience some ranking anomalies for a short period of time. Impossible to predict, but at least you can be sure there won't be any negative longer term effects.

 

Again: Nothing happens if you have a solid website with good reputation.

 

My sites are crawled daily by google, and i'm still indexed all-right, just with some 404 increased notices.

 

Still, if you stay offline for a week or more, then i don't know.

[SarisIsop]

Front page I put up on temporary cPanel is now showing the default error again.

 

Tried clearing cache and tried other browsers but no luck.

 

The file is still there in the temp cPanel.

 

If anyone has any ideas I would be grateful, but I don't want to waste a Ticket as it's not a problem to bother MDD with.

[SarisIsop]

Perhaps we can start a thread for discussion of backup plans/strategies...

 

 

 

I have started one away from this thread in Off Topic, if anyone want's to help please do so here:

 

https://forums.mddhosting.com/topic/1584-backup-plans-strategies/

[vask7]

Anyone else having SMTP Auth errors on R1 ?

 

Have opened a ticket, but not resolved yet.

[djMot]

I am on S2 and this morning ~6:00am central I find I'm back up.

 

Email is showing a curious backtracking with some email from quite a long time ago re-arriving. I use Outlook in POP3 mode so nothing is supposed to be left on the server once fetched. Other than that, there's the expected lack of new mail. The hardest part of this will now be verifying that bonafide senders that have me on a list haven't dropped me for around 72 hours of undeliverability.

 

Thanks for getting me back up and running.

[glennk]

Just want to throw out there I am on R1 with around 15 sites and everything looks good after restoration. I need to check a bit deeper but so far so good!

[cziv]

It seems that my 1st plan on S2 is ok, i don't see any (obvious) problems.

 

www and email all ok.