Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#241 LShoe

LShoe

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 24 September 2018 - 05:01 PM

I have backups done through Wordpress, of files and database, but I don't have a full cPanel backup.


  • 0

#242 MisterNeutron

MisterNeutron

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 24 September 2018 - 05:22 PM

How will the damages incurred be compensated ?

 

I am running a webshop generating quite a bit of revenue and 4 - 5 days without any sales will cost me at least US$ 5000 (direct damages).

Read the terms of service: https://www.mddhosting.com/tos.php

 

I don't think there's a web host on the planet that will compensate people for lost income due to downtime. They couldn't - a single 6-hour outage would effectively put them out of business. Does the electric company compensate businesses for lost revenue when the power is out for a couple of days?

 

Last but not least, if your site is generating that much income, do you think using a bargain shared hosting service is a smart business decision? Or should you perhaps be paying more for hosting that has more redundancy, and quicker disaster recovery? Or should you perhaps maintain two hosting services, with the ability to change your nameservers in a heartbeat, and have all of your business directed to the secondary service in case of unfortunate problems like this one?


  • 5

#243 starki

starki

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 24 September 2018 - 05:48 PM

 

And i said Google doesn't de-index anything so fast. Mine are all indexed all right.

 

I'm running a SEO agency for over 10 years, so I may share some details on this: It's pure luck how fast Google de-indexes a site if it's offline since it depends on when the site was last crawled and how fast Google processes the data to the live index. Can happen in a few hours, can take weeks till Google recognizes the site is gone. De-indexing is no penalty (although some penalties can cause de-indexing, but that's a diffent topic and doesn't matter here), Google simply doesn't want to show people an error page since that's a bad user experience.

 

The good news: Once the site is back online, it will return to/in the old positions. How fast depends on the same factors as de-indexing, the more popular the site, the better it's linked from external sites, the more often it gets crawled, the faster it will re-appear in the SERPs. Worst case downtime and a new IP may trigger a re-evaluation with a ranking rollercoaster ride for a few weeks till it settles in the old positions. Best case, the site instantly ranks like before. So, there is no permanent damage due to the downtime, but you may experience some ranking anomalies for a short period of time. Impossible to predict, but at least you can be sure there won't be any negative longer term effects.


  • 1

#244 MisterNeutron

MisterNeutron

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 24 September 2018 - 05:56 PM

 

...I may share some details on this....

 

An excellent summary. And Google doesn't give up on a site. I've noticed the Google bot hitting my sites, looking for pages that haven't existed for years. There must be some links to them out there, somewhere. It happens with surprising frequency.


  • 0

#245 ttfan

ttfan

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male
  • Location:Australia

Posted 24 September 2018 - 06:01 PM

I hope someone here can help, as Support is probably a little too busy.

 

My site (ooakforum.com) is on P1, but it's still not working after almost 12h.

 

I'm using cloudflare.

 

My domain is externally hosted, so there should be no DNS delays/
They point to name servers:
YOLANDA.NS.CLOUDFLARE.COM
MERLIN.NS.CLOUDFLARE.COM

 

I do have a dedicated IP, but I'm not sure if I need it for this.

 

I've purged the cloudflare cache, and my browser caches, it's still not working.

 

Is there something else I need to do?

 

Can anyone help?


  • 0

#246 LShoe

LShoe

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 24 September 2018 - 06:03 PM

Thank you, @Starki - that's good news indeed. I just hope my site wasn't crawled while I had just a customized "site not available" page up and an otherwise empty but live site.


  • 0

#247 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 06:06 PM

I have backups done through Wordpress, of files and database, but I don't have a full cPanel backup.

You could restore those files and databases to get your site back online - but that's all you'd have back is the site itself.

 

Read the terms of service: https://www.mddhosting.com/tos.php

 

I don't think there's a web host on the planet that will compensate people for lost income due to downtime. They couldn't - a single 6-hour outage would effectively put them out of business. Does the electric company compensate businesses for lost revenue when the power is out for a couple of days?

 

Last but not least, if your site is generating that much income, do you think using a bargain shared hosting service is a smart business decision? Or should you perhaps be paying more for hosting that has more redundancy, and quicker disaster recovery? Or should you perhaps maintain two hosting services, with the ability to change your nameservers in a heartbeat, and have all of your business directed to the secondary service in case of unfortunate problems like this one?

Anybody offline due to this that is losing money - I urge them to reach out to us and provide us with their own backups so we can get them back online now without having to wait on our backup system which is far slower than it should be.  We already have corrected the issue for backups moving forward but can't retroactively fix the old ones.

 

I hope someone here can help, as Support is probably a little too busy.

 

My site (ooakforum.com) is on P1, but it's still not working after almost 12h.

 

I'm using cloudflare.

 

My domain is externally hosted, so there should be no DNS delays/
They point to name servers:
YOLANDA.NS.CLOUDFLARE.COM
MERLIN.NS.CLOUDFLARE.COM

 

I do have a dedicated IP, but I'm not sure if I need it for this.

 

I've purged the cloudflare cache, and my browser caches, it's still not working.

 

Is there something else I need to do?

 

Can anyone help?

We already replied to your ticket.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#248 LShoe

LShoe

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 24 September 2018 - 06:11 PM

If I would have the site itself, with my backups, Mike, what would I not have (other than email)?


  • 0

#249 Rhody401

Rhody401

    Newbie

  • Members
  • Pip
  • 12 posts
  • Gender:Male
  • Location:Providence, RI USA
  • Interests:Consultant, Contractor, Beta Tester, IT Director.

Posted 24 September 2018 - 06:22 PM

Good news!  Your PHPBB forum comes up fine for me. It shows 7 registered users and 79 guests currently online from the past 20 minutes.  Maybe a browser cache?

 

The newest post seems to be from 9/12/2018, however.  But I'm not signed in, either, to see private content.

 

I hope someone here can help, as Support is probably a little too busy.

 

My site (ooakforum.com) is on P1, but it's still not working after almost 12h.

 

....

 

I do have a dedicated IP, but I'm not sure if I need it for this.

 

I've purged the cloudflare cache, and my browser caches, it's still not working.

 

Is there something else I need to do?

 

Can anyone help?


  • 0

#250 CLINECO

CLINECO

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 24 September 2018 - 07:36 PM

Does anybody know the date of the "last chance/offsite" backup?


  • 0

#251 Kevin

Kevin

    Newbie

  • Members
  • Pip
  • 15 posts

Posted 24 September 2018 - 07:52 PM

Does anybody know the date of the "last chance/offsite" backup?

"Most are only about 14 hours before the outage, one or two servers are 48 hours or so." - https://forums.mddho...cussion/?p=6923


  • 0

#252 CLINECO

CLINECO

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 24 September 2018 - 07:53 PM

"Most are only about 14 hours before the outage, one or two servers are 48 hours or so." - https://forums.mddho...cussion/?p=6923

Thanks Kevin


  • 0

#253 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 07:54 PM

Most are on September 20 - the night before the incident.  Some are on the 21st early morning.  A few on the 19th or 18th - and it varies from account to account but most are on the latest times.

 

The worst I've seen was 4 days in the past.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#254 MisterNeutron

MisterNeutron

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 24 September 2018 - 08:28 PM


My site (ooakforum.com) is on P1, but it's still not working after almost 12h.

 

It took some fairly aggressive cache-clearing in Firefox, but your forum then came up just fine - latest post is from tomorrow (ah, the mysteries of the International Date Line!).


  • 0

#255 Rhody401

Rhody401

    Newbie

  • Members
  • Pip
  • 12 posts
  • Gender:Male
  • Location:Providence, RI USA
  • Interests:Consultant, Contractor, Beta Tester, IT Director.

Posted 24 September 2018 - 08:42 PM

I'm starting to see some LIFE on one of my R1 accounts.  :)   I just received backed up (re-sent) emails from that domain from 9/17 until 2:30pm.  (so I'm assuming Phoenix backup was from about then, and I'm one that is 4 days in the past)  I can live with that :)

 

I can get into that account with FTP(and see all my files), but web browser brings me to the original cpanel "Sorry..." message. (instead of the modified one that was up earlier). I tried with a different browser and can get in, so probably just a cache.

 

Thank you! :)


  • 0

#256 vask7

vask7

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 24 September 2018 - 09:22 PM

Thanks for your commitment Michael.

 

Are we waiting for a full restore before we can access WHM?

 

Moreover, is there a way we can force redirect the default webpage back to / , and force clear the browser-cached redirection?

 

Thanks again for your efforts. 


  • 0

#257 bmolnar

bmolnar

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 24 September 2018 - 09:37 PM

Thanks for your commitment Michael.

 

Are we waiting for a full restore before we can access WHM?

 

Moreover, is there a way we can force redirect the default webpage back to / , and force clear the browser-cached redirection?

 

Thanks again for your efforts. 

I have no way to test this since my site on S3 is still down, but you should be able to put this in the .htaccess file in /www

<FilesMatch "\.(css|flv|gif|htm|html|ico|jpe|jpeg|jpg|js|png|pdf|swf|txt)$">
	<IfModule mod_expires.c>
		ExpiresActive Off
	</IfModule>
	<IfModule mod_headers.c>
		FileETag None
		Header unset ETag
		Header unset Pragma
		Header unset Cache-Control
		Header unset Last-Modified
		Header set Pragma "no-cache"
		Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate"
		Header set Expires "Mon, 10 Apr 1972 00:00:00 GMT"
	</IfModule>
</FilesMatch>

On a side-note, from any server that hasn't been restored yet, are any of you able to login to Cpanel or do we have to wait?  Thanks


  • 0

#258 spocklikesspores

spocklikesspores

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 24 September 2018 - 09:49 PM

Did I miss something or was the last ETA/progress posting from MDD early this AM? By now there should be additional information...is it being posted somewhere that I'm not tuned in to?


  • 0

#259 bmolnar

bmolnar

    Newbie

  • Members
  • Pip
  • 7 posts

Posted 24 September 2018 - 09:51 PM

Did I miss something or was the last ETA/progress posting from MDD early this AM? By now there should be additional information...is it being posted somewhere that I'm not tuned in to?

 

Updates from staff are being posted in another thread:  https://forums.mddhosting.com/topic/1582-major-outage-092118-09242018/  This thread is just for discussion


  • 0

#260 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 10:03 PM

Thanks for your commitment Michael.

 

Are we waiting for a full restore before we can access WHM?

 

Moreover, is there a way we can force redirect the default webpage back to / , and force clear the browser-cached redirection?

 

Thanks again for your efforts. 

You should have access to WHM but if you don't - open a ticket and we can reset permissions.

 

Unfortunately the cgi-sys folder isn't actually loaded from your account so you can't do anything to affect it.

 

 

Updates from staff are being posted in another thread:  https://forums.mddhosting.com/topic/1582-major-outage-092118-09242018/  This thread is just for discussion

Correct - we split them out because it was becoming impossible for people to keep up with the status updates we provide - but we didn't want to kill the conversation either.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users