Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#321 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 02:06 PM

Once my sites (s3) come back up, should I be able to immediately access them through my FTP client or cPanel? Just looking for a more surefire way to tell if I'm up and running other than hitting F5 on the sites all day.

Hitting F5 will reload the error page. Make sure you’re trying your domain. 


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#322 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 02:07 PM

 

That was the issue for our "admin" site. All our client sites had issues with MainWP. Renaming those plugin folders in cPanel File Manager brought the rest of our sites back online. (and I'm pretty sure it was MainWP that required PHP 7 on the admin site.....) 

 

Just trying to help... I would have much preferred being able to fix the issue myself without having to bother you guys.. you got a lot on your plate right now. 

No worries. Much appreciated. 


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#323 Big Dan

Big Dan

    Member

  • Members
  • PipPip
  • 46 posts
  • Gender:Male

Posted 25 September 2018 - 02:25 PM

 

This would be excellent advice for everyone.  However, I'm also going to add that if you're using WHM (I haven't seen the options for Cpanel), there's a backup configuration that can be configured to automatically send daily server backups to an Amazon S3 account, then configure lifecycle rules, versioning rules, etc on the S3 side so you have continuous backups of your hosting reseller or VPS account remotely.  I have backups going back a solid 12 months.  These are daily, weekly and monthly.  I then have lifecycle rules that push all backups over 30 days to Amazon Glacier and expire them out after one year.  If you think it takes hours to set this up, it took me a grand total of about 15 minutes.  That 15 min was well spent knowing my backups are transferred offsite every morning at 2am.

 

Always, always have your own backups and use some scripting to try and automate the process to off-site your own backups so you don't have to remember to download your own server backups.  At the end of the day, we're all human.

 

Thanks for the suggestion Kev. I have daily backups to S3 on my VPS with MDD, which ironically stayed online, but wasn't aware resellers had the remote backup option. 


  • 0

#324 Big Dan

Big Dan

    Member

  • Members
  • PipPip
  • 46 posts
  • Gender:Male

Posted 25 September 2018 - 02:31 PM

Just wondering if anyone is experiencing a lag with services being accessible/live after server restoration is complete.  I swear I'm not impatient, but I saw r4 was recently restored but it seems like the sites are slowly flickering on, and I'm still unable to log into WHM from the MDD dashboard.  Just wondering if I need to sit tight or open a ticket?  Also, does anyone know if emails during downtime will eventually come through or are they lost for good?

Open a ticket. I had to in order to restore reseller access to an account on R4. cPanel login worked fine with the same credentials just a blip in the restore once Tim tinkered all came back 100%. 


  • 0

#325 digibread

digibread

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 25 September 2018 - 02:32 PM

 

Hey Russ, 

 

If you're using MainWP, try renaming the plugin folders for mainWP child and MainWP child reports. They tracked that down to being the issue my sites were having. 

 

jim

Thank you Jim, I do use MainWP but it turned out to be UpdraftPlus. All is good now. Hope the same for you. Best, Russ


  • 0

#326 digibread

digibread

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 25 September 2018 - 02:38 PM

OK, I'm 100% back up, a few hiccups and a little work but all in all my customers remained cool throughout the process.

 

Thank you Michael and Team at MDD Hosting. You guys have always offered the best service and support in the industry and I can only imagine after an event like this you will be rock solid for the years to come. I'm going to remain a loyal customer and not going anywhere.

 

As a matter a fact I know that the costs of this for you and your company is probably beyond devastating. I have setup a GoFundMe to try and help out. A small business is made up of people and families with bills and obligations.

 

This is my plea to fellow loyal customers. I've been hosting websites for 18 years and I've experienced the worst the industry has to offer.
MDD Hosting has been amazing and finding a company with the level of support is very hard. Michael is a pillar of business ethics and I trust what he says. All that being said, I hope you consider helping out.
 
Thanks, Russ

  • 0

#327 Avatar

Avatar

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 25 September 2018 - 02:51 PM

If the catastrophic failure occurred on Friday, 925/2018 at approximately 5:00PM EST, why -- once our email accounts were restored early Monday morning --  did we not eventually receive all of the email which was sent to our domain email addresses on Saturday and Sunday?


  • 0

#328 chris.holmes

chris.holmes

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 25 September 2018 - 02:52 PM

OK, dumb question time. I can get into my site through cPanel and FTP, and I can access them through browsers on other computers but not my primary. I've restarted the browser, cleared cache and cookies and still hit the error pages. What minor thing am I missing?


  • 0

#329 comvox

comvox

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 25 September 2018 - 02:52 PM

I was astonished to read the following statement on the locked outage forum today:  "If anybody has a backup of their own - we can use that to get you online immediately. "  I didn't know that. I wish it had been said in the MDDHosting emails about the outage. I meticulously keep a backup.  Well, I'm on S3 and should be restored by late tonight or very early a.m. tomorrow anyway, so I don't see the point of sending in my backup now. It would just add unnecessary work to an already overburdened staff.  And I still have confidence in MDDHosting. But talk about rubbing salt in a wound....


  • 0

#330 Bookproc

Bookproc

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 25 September 2018 - 03:05 PM

If the catastrophic failure occurred on Friday, 925/2018 at approximately 5:00PM EST, why -- once our email accounts were restored early Monday morning --  did we not eventually receive all of the email which was sent to our domain email addresses on Saturday and Sunday?

You should get the ones that are/were automatically still being resent by the senders' mail systems once your accounts came back up. Until your domain was restored, those addresses were unavailable, so there was nowhere for the sent mail to go. On Saturday and Sunday, most of the people who tried to send you email will have received bounce messages. Depending on their mail system, the bounce message may have indicated that the sending software would retry every X hours for a period of Y hours before giving up. Often the Y is 24 or 48 hours. For systems that retry sending, if all of the retries occurred before your account was restored, the sending software will have informed the senders that the email address is unreachable and the bounce is permanent. If they retried after your accounts were restored, you should still get the email. If the bounce was permanent, the sender will have to send the email again for it to reach you.


  • 0

#331 LazyDodo

LazyDodo

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 25 September 2018 - 03:20 PM

You should get the ones that are/were automatically still being resent by the senders' mail systems once your accounts came back up. Until your domain was restored, those addresses were unavailable, so there was nowhere for the sent mail to go. On Saturday and Sunday, most of the people who tried to send you email will have received bounce messages. Depending on their mail system, the bounce message may have indicated that the sending software would retry every X hours for a period of Y hours before giving up. Often the Y is 24 or 48 hours. For systems that retry sending, if all of the retries occurred before your account was restored, the sending software will have informed the senders that the email address is unreachable and the bounce is permanent. If they retried after your accounts were restored, you should still get the email. If the bounce was permanent, the sender will have to send the email again for it to reach you.

 

Didn't quite work that way, i send a test email from a non MDD account before my server got restored, there was no bounce, no retry, just a single terse 'this is not happening my friend'

 

This message was created automatically by mail delivery software.


A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:


  xxxxx@myemail.com
    host p1.supportedns.com [173.248.191.179]
    SMTP error from remote mail server after RCPT TO:<xxxxx@myemail.com>:
    550-Please turn on SMTP Authentication in your mail client.
    550-sending_host.com [xxx.xxx.xxx.xxx]:34282 is not permitted to relay
    550 through this server without authentication.

  • 0

#332 KevinD872

KevinD872

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:NW Indiana
  • Interests:Drums & music, motorcycles, computers, programming and gadgets.

Posted 25 September 2018 - 03:42 PM

What is the proper way to back up all cPanel-related site settings?  Files>Backup Wizard or Jetbackup>Full Account Backups or other?  When I try going into Jetbackup it says I don't have permission to access the page.

 

Pretty much anything that I can change in cPanel, I want to be able to restore if ever needed.  What's the best way to do this?


  • 0

#333 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 155 posts
  • Gender:Not Telling

Posted 25 September 2018 - 03:50 PM

What is the proper way to back up all cPanel-related site settings?  Files>Backup Wizard or Jetbackup>Full Account Backups or other?  When I try going into Jetbackup it says I don't have permission to access the page.

 

Pretty much anything that I can change in cPanel, I want to be able to restore if ever needed.  What's the best way to do this?

 

This may help:

 

https://forums.mddho...ans-strategies/


  • 1

#334 Shell

Shell

    Grand Poobahess

  • Members
  • Pip
  • 4 posts
  • Gender:Not Telling
  • Location:Metro Vancouver, BC Canada

Posted 25 September 2018 - 03:52 PM

I also have been a longtime customer (so long, I can't remember when I started) and can see where the repercussions here could be significant.  I haven't commented to date but rather have been sitting back reading and waiting.  It's been hard.  I'm lucky that 3 of my main clients are on Office 365 so they haven't experienced any email issues.  Email is critical to them (insurance agency, chartered accountants, business advisors).  The server my sites are on, isn't up yet. I'm on the run tomorrow so I'm hoping no issues......................... (Pacific Time here) 

 

I have a reseller account and have had since the beginning but don't do this as a business.  I'm a web designer by trade but have offered hosting to my clients up until the past 2 years as a service to make their lives easier (no need to understand about that aspect of your website) and also easier for me as familiar with one system (you only have to try and work with our main telephone & internet provider's system to truly understand frustration).  I've slowly paired down the amount of accounts that I had to the ones that I work with on a regular basis.

 

However, I have now lost one of my main three.  So the loss of the hosting - not a big deal.  It's the loss of my standing with the company that is.  They're ok, as I mentioned re their emails and the site down isn't a huge issue - it is the length of time that it has been down and the inability for me to address that quickly.  And I put faith in a system that I shouldn't have (as many have) that even tho I had backups - the majority of them are on the darn server.  The other backups - those of sites that I have recently redone - well they are on my backup harddrive which is in storage.  (We sold our house and are living in our RV as the deal for the house we wanted collapsed.  Now that's a story for another day.)  They understand but in their eyes there is a failure here that goes beyond the down website.  I take full responsibility for my end of things and I can get the harddrive (but need help and a day to get it) however it seemed pointless given the time frame.  For example, I thought the system would be back up by Monday and therefore not a huge concern.  The fact that R3 is scheduled for restore 3am Wednesday is definitely a factor.

 

So, they will move onto different hosting and a different web/social media person and I've learned a lesson.  My other 2 main clients are long-term and are weathering this out with patience.  My smaller clients are not happy - they aren't on Office 365.  Mostly with me as I'm the one who set their site up here so this falls back to me.

 

And I get the compassion and empathy behind the GoFundMe page.  I think it's an admirable thing to do and I will consider donating............. once I've reconciled & accepted my losses.

 

MDD Hosting hasn't lost me as a client - I think this has been a huge learning experience for us all.  I'll stay (besides I'm definitely small potatoes lol!) because I believe in their genuineness and sincere efforts to get everyone's sites back up.  In all the time I've been with this company I have appreciated their support, the attentiveness to security and the commitment to their clients. 

 

I'd rather give my, albeit small, business to a company that operates much along the lines I do. Where you can talk with the owner and not some anonymous service clerk. It's why I've stayed for the years I have.  I was mad - not going to mince words about that - but I was about as mad at myself as I was at MDD. 

 

Ok now, nothing like time and a little wine ;-)

 

So my thanks to all that are working so diligently on this. (ROFL! had to edit that from "******" this as it was ***** out :-D   hahahahah ok, I give up - you guess what I wrote! ;-p  )

 

Shelley

 

 

 

 

OK, I'm 100% back up, a few hiccups and a little work but all in all my customers remained cool throughout the process.

 

Thank you Michael and Team at MDD Hosting. You guys have always offered the best service and support in the industry and I can only imagine after an event like this you will be rock solid for the years to come. I'm going to remain a loyal customer and not going anywhere.

 

As a matter a fact I know that the costs of this for you and your company is probably beyond devastating. I have setup a GoFundMe to try and help out. A small business is made up of people and families with bills and obligations.

 

This is my plea to fellow loyal customers. I've been hosting websites for 18 years and I've experienced the worst the industry has to offer.
MDD Hosting has been amazing and finding a company with the level of support is very hard. Michael is a pillar of business ethics and I trust what he says. All that being said, I hope you consider helping out.
 
Thanks, Russ

 


  • 1

#335 Shell

Shell

    Grand Poobahess

  • Members
  • Pip
  • 4 posts
  • Gender:Not Telling
  • Location:Metro Vancouver, BC Canada

Posted 25 September 2018 - 03:54 PM

I was astonished to read the following statement on the locked outage forum today:  "If anybody has a backup of their own - we can use that to get you online immediately. "  I didn't know that. I wish it had been said in the MDDHosting emails about the outage. I meticulously keep a backup.  Well, I'm on S3 and should be restored by late tonight or very early a.m. tomorrow anyway, so I don't see the point of sending in my backup now. It would just add unnecessary work to an already overburdened staff.  And I still have confidence in MDDHosting. But talk about rubbing salt in a wound....

 

Yes I agree - I was behind on reading ALL the messages in the two areas posted.  I do believe it would have been a good announcement to make with proper subject heading and pinned.  Maybe in caps ;-)


  • 0

#336 Shell

Shell

    Grand Poobahess

  • Members
  • Pip
  • 4 posts
  • Gender:Not Telling
  • Location:Metro Vancouver, BC Canada

Posted 25 September 2018 - 04:10 PM

Question (as a sidebar from my post above)

 

My Office 365 clients obviously have had the MX configured for this.  I need to be prepared if I have to redo that and advise their IT people so it can be verified on their end.

 

Is there an expectation that the MX will default to original settings?

 

(Asking here instead of Ticket as no issue yet and helpful info for others........... I hope)

 

Shelley


  • 0

#337 KevinD872

KevinD872

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:NW Indiana
  • Interests:Drums & music, motorcycles, computers, programming and gadgets.

Posted 25 September 2018 - 04:19 PM

I've been back up for a couple hours or so, roughly (S3).  So far everything is intact and working.  Had to re-enter a password or two into Android devices but all is good.  I'm only mentioning this to hopefully give some peace of mind to others still waiting that at least for me, everything is still working as before.


  • 1

#338 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 04:22 PM

 

Yes I agree - I was behind on reading ALL the messages in the two areas posted.  I do believe it would have been a good announcement to make with proper subject heading and pinned.  Maybe in caps ;-)

We tweeted it a few times and I do apologize but I sort of assumed, incorrectly, that anybody with their own backups would ask us to restore them.

 

The servers have been up since Friday - within a few hours of the outage - the only thing we've been doing this whole time is restoring our own backups.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#339 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 04:23 PM

Question (as a sidebar from my post above)

 

My Office 365 clients obviously have had the MX configured for this.  I need to be prepared if I have to redo that and advise their IT people so it can be verified on their end.

 

Is there an expectation that the MX will default to original settings?

 

(Asking here instead of Ticket as no issue yet and helpful info for others........... I hope)

 

Shelley

MX Records are saved in the backups we have and should be restored as they were.  I actually went into our backup system and got the Outlook365 MX Records for one of our clients yesterday.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#340 digibread

digibread

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 25 September 2018 - 04:30 PM

If the catastrophic failure occurred on Friday, 925/2018 at approximately 5:00PM EST, why -- once our email accounts were restored early Monday morning --  did we not eventually receive all of the email which was sent to our domain email addresses on Saturday and Sunday?

Hey Avatar, my understanding is that properly configured mail servers when sending will keep on attempting to deliver the mail for an extended period (up to 10 days in my experience) if it does give up the recipient will receive a undeliverable message so that they know to try again. This way mail isn't lost in space when stuff like this happens. I hope that helps. Russ


  • 0




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users