Jump to content
MDDHosting Forums

Major Outage - 09/21/18+ - Client Discussion


KevinD872

Recommended Posts

OK, I'm 100% back up, a few hiccups and a little work but all in all my customers remained cool throughout the process.

 

Thank you Michael and Team at MDD Hosting. You guys have always offered the best service and support in the industry and I can only imagine after an event like this you will be rock solid for the years to come. I'm going to remain a loyal customer and not going anywhere.

 

As a matter a fact I know that the costs of this for you and your company is probably beyond devastating. I have setup a GoFundMe to try and help out. A small business is made up of people and families with bills and obligations.

 

This is my plea to fellow loyal customers. I've been hosting websites for 18 years and I've experienced the worst the industry has to offer.
MDD Hosting has been amazing and finding a company with the level of support is very hard. Michael is a pillar of business ethics and I trust what he says. All that being said, I hope you consider helping out.
Thanks, Russ
Link to comment
Share on other sites

I was astonished to read the following statement on the locked outage forum today: "If anybody has a backup of their own - we can use that to get you online immediately. " I didn't know that. I wish it had been said in the MDDHosting emails about the outage. I meticulously keep a backup. Well, I'm on S3 and should be restored by late tonight or very early a.m. tomorrow anyway, so I don't see the point of sending in my backup now. It would just add unnecessary work to an already overburdened staff. And I still have confidence in MDDHosting. But talk about rubbing salt in a wound....

Link to comment
Share on other sites

If the catastrophic failure occurred on Friday, 925/2018 at approximately 5:00PM EST, why -- once our email accounts were restored early Monday morning -- did we not eventually receive all of the email which was sent to our domain email addresses on Saturday and Sunday?

You should get the ones that are/were automatically still being resent by the senders' mail systems once your accounts came back up. Until your domain was restored, those addresses were unavailable, so there was nowhere for the sent mail to go. On Saturday and Sunday, most of the people who tried to send you email will have received bounce messages. Depending on their mail system, the bounce message may have indicated that the sending software would retry every X hours for a period of Y hours before giving up. Often the Y is 24 or 48 hours. For systems that retry sending, if all of the retries occurred before your account was restored, the sending software will have informed the senders that the email address is unreachable and the bounce is permanent. If they retried after your accounts were restored, you should still get the email. If the bounce was permanent, the sender will have to send the email again for it to reach you.

Link to comment
Share on other sites

You should get the ones that are/were automatically still being resent by the senders' mail systems once your accounts came back up. Until your domain was restored, those addresses were unavailable, so there was nowhere for the sent mail to go. On Saturday and Sunday, most of the people who tried to send you email will have received bounce messages. Depending on their mail system, the bounce message may have indicated that the sending software would retry every X hours for a period of Y hours before giving up. Often the Y is 24 or 48 hours. For systems that retry sending, if all of the retries occurred before your account was restored, the sending software will have informed the senders that the email address is unreachable and the bounce is permanent. If they retried after your accounts were restored, you should still get the email. If the bounce was permanent, the sender will have to send the email again for it to reach you.

 

Didn't quite work that way, i send a test email from a non MDD account before my server got restored, there was no bounce, no retry, just a single terse 'this is not happening my friend'

 

This message was created automatically by mail delivery software.


A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:


  xxxxx@myemail.com
    host p1.supportedns.com [173.248.191.179]
    SMTP error from remote mail server after RCPT TO:<xxxxx@myemail.com>:
    550-Please turn on SMTP Authentication in your mail client.
    550-sending_host.com [xxx.xxx.xxx.xxx]:34282 is not permitted to relay
    550 through this server without authentication.
Link to comment
Share on other sites

What is the proper way to back up all cPanel-related site settings? Files>Backup Wizard or Jetbackup>Full Account Backups or other? When I try going into Jetbackup it says I don't have permission to access the page.

 

Pretty much anything that I can change in cPanel, I want to be able to restore if ever needed. What's the best way to do this?

Link to comment
Share on other sites

What is the proper way to back up all cPanel-related site settings? Files>Backup Wizard or Jetbackup>Full Account Backups or other? When I try going into Jetbackup it says I don't have permission to access the page.

 

Pretty much anything that I can change in cPanel, I want to be able to restore if ever needed. What's the best way to do this?

 

This may help:

 

https://forums.mddhosting.com/topic/1584-backup-plans-strategies/

  • Upvote 1
Link to comment
Share on other sites

I also have been a longtime customer (so long, I can't remember when I started) and can see where the repercussions here could be significant. I haven't commented to date but rather have been sitting back reading and waiting. It's been hard. I'm lucky that 3 of my main clients are on Office 365 so they haven't experienced any email issues. Email is critical to them (insurance agency, chartered accountants, business advisors). The server my sites are on, isn't up yet. I'm on the run tomorrow so I'm hoping no issues......................... (Pacific Time here)

 

I have a reseller account and have had since the beginning but don't do this as a business. I'm a web designer by trade but have offered hosting to my clients up until the past 2 years as a service to make their lives easier (no need to understand about that aspect of your website) and also easier for me as familiar with one system (you only have to try and work with our main telephone & internet provider's system to truly understand frustration). I've slowly paired down the amount of accounts that I had to the ones that I work with on a regular basis.

 

However, I have now lost one of my main three. So the loss of the hosting - not a big deal. It's the loss of my standing with the company that is. They're ok, as I mentioned re their emails and the site down isn't a huge issue - it is the length of time that it has been down and the inability for me to address that quickly. And I put faith in a system that I shouldn't have (as many have) that even tho I had backups - the majority of them are on the darn server. The other backups - those of sites that I have recently redone - well they are on my backup harddrive which is in storage. (We sold our house and are living in our RV as the deal for the house we wanted collapsed. Now that's a story for another day.) They understand but in their eyes there is a failure here that goes beyond the down website. I take full responsibility for my end of things and I can get the harddrive (but need help and a day to get it) however it seemed pointless given the time frame. For example, I thought the system would be back up by Monday and therefore not a huge concern. The fact that R3 is scheduled for restore 3am Wednesday is definitely a factor.

 

So, they will move onto different hosting and a different web/social media person and I've learned a lesson. My other 2 main clients are long-term and are weathering this out with patience. My smaller clients are not happy - they aren't on Office 365. Mostly with me as I'm the one who set their site up here so this falls back to me.

 

And I get the compassion and empathy behind the GoFundMe page. I think it's an admirable thing to do and I will consider donating............. once I've reconciled & accepted my losses.

 

MDD Hosting hasn't lost me as a client - I think this has been a huge learning experience for us all. I'll stay (besides I'm definitely small potatoes lol!) because I believe in their genuineness and sincere efforts to get everyone's sites back up. In all the time I've been with this company I have appreciated their support, the attentiveness to security and the commitment to their clients.

 

I'd rather give my, albeit small, business to a company that operates much along the lines I do. Where you can talk with the owner and not some anonymous service clerk. It's why I've stayed for the years I have. I was mad - not going to mince words about that - but I was about as mad at myself as I was at MDD.

 

Ok now, nothing like time and a little wine ;-)

 

So my thanks to all that are working so diligently on this. (ROFL! had to edit that from "******" this as it was ***** out :-D hahahahah ok, I give up - you guess what I wrote! ;-p )

 

Shelley

 

 

 

 

OK, I'm 100% back up, a few hiccups and a little work but all in all my customers remained cool throughout the process.

 

Thank you Michael and Team at MDD Hosting. You guys have always offered the best service and support in the industry and I can only imagine after an event like this you will be rock solid for the years to come. I'm going to remain a loyal customer and not going anywhere.

 

As a matter a fact I know that the costs of this for you and your company is probably beyond devastating. I have setup a GoFundMe to try and help out. A small business is made up of people and families with bills and obligations.

 

This is my plea to fellow loyal customers. I've been hosting websites for 18 years and I've experienced the worst the industry has to offer.
MDD Hosting has been amazing and finding a company with the level of support is very hard. Michael is a pillar of business ethics and I trust what he says. All that being said, I hope you consider helping out.
Thanks, Russ

 

  • Upvote 1
Link to comment
Share on other sites

I was astonished to read the following statement on the locked outage forum today: "If anybody has a backup of their own - we can use that to get you online immediately. " I didn't know that. I wish it had been said in the MDDHosting emails about the outage. I meticulously keep a backup. Well, I'm on S3 and should be restored by late tonight or very early a.m. tomorrow anyway, so I don't see the point of sending in my backup now. It would just add unnecessary work to an already overburdened staff. And I still have confidence in MDDHosting. But talk about rubbing salt in a wound....

 

Yes I agree - I was behind on reading ALL the messages in the two areas posted. I do believe it would have been a good announcement to make with proper subject heading and pinned. Maybe in caps ;-)

Link to comment
Share on other sites

Question (as a sidebar from my post above)

 

My Office 365 clients obviously have had the MX configured for this. I need to be prepared if I have to redo that and advise their IT people so it can be verified on their end.

 

Is there an expectation that the MX will default to original settings?

 

(Asking here instead of Ticket as no issue yet and helpful info for others........... I hope)

 

Shelley

Link to comment
Share on other sites

I've been back up for a couple hours or so, roughly (S3). So far everything is intact and working. Had to re-enter a password or two into Android devices but all is good. I'm only mentioning this to hopefully give some peace of mind to others still waiting that at least for me, everything is still working as before.

  • Upvote 1
Link to comment
Share on other sites

 

Yes I agree - I was behind on reading ALL the messages in the two areas posted. I do believe it would have been a good announcement to make with proper subject heading and pinned. Maybe in caps ;-)

We tweeted it a few times and I do apologize but I sort of assumed, incorrectly, that anybody with their own backups would ask us to restore them.

 

The servers have been up since Friday - within a few hours of the outage - the only thing we've been doing this whole time is restoring our own backups.

Link to comment
Share on other sites

Question (as a sidebar from my post above)

 

My Office 365 clients obviously have had the MX configured for this. I need to be prepared if I have to redo that and advise their IT people so it can be verified on their end.

 

Is there an expectation that the MX will default to original settings?

 

(Asking here instead of Ticket as no issue yet and helpful info for others........... I hope)

 

Shelley

MX Records are saved in the backups we have and should be restored as they were. I actually went into our backup system and got the Outlook365 MX Records for one of our clients yesterday.

Link to comment
Share on other sites

If the catastrophic failure occurred on Friday, 925/2018 at approximately 5:00PM EST, why -- once our email accounts were restored early Monday morning -- did we not eventually receive all of the email which was sent to our domain email addresses on Saturday and Sunday?

Hey Avatar, my understanding is that properly configured mail servers when sending will keep on attempting to deliver the mail for an extended period (up to 10 days in my experience) if it does give up the recipient will receive a undeliverable message so that they know to try again. This way mail isn't lost in space when stuff like this happens. I hope that helps. Russ

Link to comment
Share on other sites

OK, dumb question time. I can get into my site through cPanel and FTP, and I can access them through browsers on other computers but not my primary. I've restarted the browser, cleared cache and cookies and still hit the error pages. What minor thing am I missing?

Hey Chris, if you're on windows you might try a "command prompt" and type in this without the quotes. "ipconfig /flushdns" a restart will likely do the same. I hope that helps.

Link to comment
Share on other sites

Hey Chris, if you're on windows you might try a "command prompt" and type in this without the quotes. "ipconfig /flushdns" a restart will likely do the same. I hope that helps.

 

if you behind a router rebooting the router sometimes resolves this.

 

another you can do is just change your ISP DNS in your local PC network settings to like google DNS or opendns

and see if you can resolve your site on your machine.

 

after 72 hours you can go back to your defaults

 

 

if you dont make any changes..only thing that is gonna work is

DNS propagation time takes 24 to 72 hours to resolve world wide when a change is made.

 

dont worry nothing wrong with your pc.

 

  • Upvote 1
Link to comment
Share on other sites

thank you for the quick response.

 

I have to take the motorhome - currently living quarters - in for service tomorrow (2 month delay for appt) and was stressing a bit that I'd have to deal with MX entries for a couple of clients. Ok, Freaking a bit to be honest

 

Shelley

 

 

MX Records are saved in the backups we have and should be restored as they were. I actually went into our backup system and got the Outlook365 MX Records for one of our clients yesterday.

Link to comment
Share on other sites

Is S3 fully restored? My site is up, but we use Shipstation to import our orders for shipping labels. Shipstation says that the connection to my website's server is fine. However, I have 3 new orders that should have imported that haven't. Everything worked fine on Thursday. I tried deleting and re-uploading and re-installing the plug-in that the site uses to communicate with Shipstation, but no avail. Shipstation says that orders should be importing from my site as they are showing no errors to the connection, but they are not.

Any ideas before I open a ticket?

Link to comment
Share on other sites

Is S3 fully restored? My site is up, but we use Shipstation to import our orders for shipping labels. Shipstation says that the connection to my website's server is fine. However, I have 3 new orders that should have imported that haven't. Everything worked fine on Thursday. I tried deleting and re-uploading and re-installing the plug-in that the site uses to communicate with Shipstation, but no avail. Shipstation says that orders should be importing from my site as they are showing no errors to the connection, but they are not.

Any ideas before I open a ticket?

Not off hand - does it require any custom ports?

Link to comment
Share on other sites

I'm on S2, using Joomla for all my sites. S2 is completely restored. All sites seems to be working. I haven't checked all components in Joomla but Akeeba backup isn't working for any site. It complains about missing angie.jpa file. I also noticed that my disk usage on my account is 1.8 Gig rather than the 4 Gig before this outage. How strange? I've already open a ticket and Mike has restored my account but same symptoms. No big deal, I was able to grab the angie.jpa file from my local xampp install and now Akeep Backup is working but it seems off that perhaps not all files are restored? Perhaps some file extension types are not included as backup? I don't know whether anyone else is having similar issues?

Link to comment
Share on other sites

I've just lost almost 2 days worth of emails!!

 

My account on s3 was mostly recovered early monday morning to the extent that at least emails worked. Recovery was done manually from a backup file with most recent data as of 9/21.

 

The ticket for that recovery was flagged as - underlined part added by MDD.

 

#657811 - Create new/empty account - Do not restore over this account.
#657811 - Create new/empty account - Do not restore over this account.
As of right now all emails since the restore and now (almost 2 days worth) are gone. It appears my account was restored again as other uploaded files are gone too.
In fact I just received an email that recovery on s3 is complete.
I had my doubts about staying, even inquired about the cancellation process (ticket #811489 - still awaiting response). But this really is making that decision easier.
@Mike, unless you're going to be offering some serious credits for the outage I have a hard time justifying a reason to stay.
Link to comment
Share on other sites

 

I've just lost almost 2 days worth of emails!!

 

My account on s3 was mostly recovered early monday morning to the extent that at least emails worked. Recovery was done manually from a backup file with most recent data as of 9/21.

 

The ticket for that recovery was flagged as - underlined part added by MDD.

 

#657811 - Create new/empty account - Do not restore over this account.
#657811 - Create new/empty account - Do not restore over this account.
As of right now all emails since the restore and now (almost 2 days worth) are gone. It appears my account was restored again as other uploaded files are gone too.
In fact I just received an email that recovery on s3 is complete.
I had my doubts about staying, even inquired about the cancellation process (ticket #811489 - still awaiting response). But this really is making that decision easier.
@Mike, unless you're going to be offering some serious credits for the outage I have a hard time justifying a reason to stay.

 

 

i feel your pain as a customer myself i feel the aggravation and the desire to split to another host

but yanno your gonna have some issues no matter what shared hosting provider you go to and the sad thing is

you never know if the next host is just a reseller host for someones main business which is even MORE of a headache

because you ll submit a ticket then they have to submit a ticket and and and....ya not worth it..

 

 

it all comes down to this..

it is our responsibility to keep our own backups of our precious files and keep our sites tip top shape

our HOST supplies a backup out of curiosity. only a Managed hosting provider is going to watch your site

and do backups and that is a heavy cost and not cheap per month either. (i rather own my own dedicated machine over that but even then its a work load)

 

you can set cron jobs to do hourly/daily/weekly/monthly backups..

you also want to keep your backups OFF your main host..

 

MDD is a small business they have grown alot since iv been with them..

in my honest opinion i rather stay then go. they watch out for you closer then some high end top tier host provider

that gives you some lame response with EVERYTHING you ask about..

now god forbid but if this happens again im sure alot of us is going to leave. that just means pure carelessness / laziness

 

i dont work for MDD im just a customer sharing my 2 cents.

Link to comment
Share on other sites

 

I've just lost almost 2 days worth of emails!!

 

My account on s3 was mostly recovered early monday morning to the extent that at least emails worked. Recovery was done manually from a backup file with most recent data as of 9/21.

 

The ticket for that recovery was flagged as - underlined part added by MDD.

 

#657811 - Create new/empty account - Do not restore over this account.
#657811 - Create new/empty account - Do not restore over this account.
As of right now all emails since the restore and now (almost 2 days worth) are gone. It appears my account was restored again as other uploaded files are gone too.
In fact I just received an email that recovery on s3 is complete.
I had my doubts about staying, even inquired about the cancellation process (ticket #811489 - still awaiting response). But this really is making that decision easier.
@Mike, unless you're going to be offering some serious credits for the outage I have a hard time justifying a reason to stay.

 

 

As I told you in the ticket - initially you wanted it overwritten and then you asked for it not to be - so you ended up on both lists. Admittedly we have to go out of our way to overwrite your account - so when we saw it on that list we did it. Did we ****** up? Yes. It's worth noting that we aren't restoring your accounts from our support ticket system - but from a restoration interface on the server itself that does not in any way reference the ticket system. If we were to reference the ticket system for every restore - we'd have only a few hundred accounts online at this time instead of many thousands.

 

The automated system did not overwrite your account - we manually did it. We checked the list of accounts to overwrite and then performed the overwrites. We honestly didn't consider that an account could be on both lists. It's an oversight - but one that won't affect anyone else because I checked and there are no other duplicates. Now that we're aware of this potential issue it's something we can avoid.

 

Did we mess up? Absolutely - and I'm sorry for that.

 

http://www.screen-shot.net/2018-09-25_20-41-27.png

 

Hi Mike, ETA for S4 full restoration?

 

You wrote:

"Due to a unexpected reboot of the backup server we are going to have to swap s5 and s4 so we can not lose progress made towards s4's restoration"

 

What does it means?

It means that we were copying the file system image for the backup when the backup server crashed. We can resume it - but we actually have to start the restore process on the other server to get the resume token. In short - we moved on to backing up another server as not to lose the time we already sank into backing up S4.

 

We're looking to be ahead of the total ETA and are hoping to have everybody online by morning.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...