Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#41 cziv

cziv

    Member

  • Members
  • PipPip
  • 52 posts
  • Gender:Male

Posted 23 September 2018 - 07:20 AM

Actually it is NOT showing Cpanel but "Your Connection is NOT SAFE,  the owner has not ... bla bla bla". Cpanel shows IF and After you select to move on accepting a few things.

 

Firefox that is.


  • 0

#42 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 07:31 AM

Actually it is NOT showing Cpanel but "Your Connection is NOT SAFE,  the owner has not ... bla bla bla". Cpanel shows IF and After you select to move on accepting a few things.
 
Firefox that is.

Unfortunately there's no way to avoid the SSL error short of installing an SSL on the server for every client domain - and without the client domains being restored - we have no way of having a full list of those domains. Furthermore if we did try to do this - when we do go to restore accounts it's going to create a lot of problems and break things.

I wish there was a good solution just like I wish this incident never happened and I wish we were back online already 100%. Unfortunately I can wish all I want and that doesn't change reality - but we are working to restore services as quickly as we can. We're unfortunately limited by the hardware that holds the data presently and we're working on overcoming that.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#43 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 154 posts
  • Gender:Not Telling

Posted 23 September 2018 - 07:40 AM

Thank you for the Email Michael.

 

I knew you would be working flat out to get this sorted, it's gone wrong but I feel comforted to know that you are still on the ball.

 

So pleased also to read that you will never sell up and that hopefully MDD will keep it's values as a family business when your Sons are old enough.

 

Yes I think you are right, you will undoubtedly lose some customers over this but If it helps, I will be staying with MDD. You have always looked after me and my forum members, even though we are just small fish.

 

As for the money you will lose, I have only just renewed for a year but if you was to offer one of your existing customer discounts I would be happy to add enough for another year if it helps.

 

I still have faith in MDD.


  • 0

#44 cziv

cziv

    Member

  • Members
  • PipPip
  • 52 posts
  • Gender:Male

Posted 23 September 2018 - 07:42 AM

Michael, i have been with you for 4 years now. I have nothing but good words for you all.

 

Having said that, what i do not like is your backup strategy. I do remember an other instance with you during those 4 years that i was NOT able to actually restore a backup from your backup pool (for some reason you said it was not working. I believe it is still in my old tickets with you).

 

So if your current backup for some reason was corrupted, you would be actually out of business. Do you realize that ?

 

Personally i do a manual Cpanel full backup every Monday (usually) for my two plans and i have automated an ftp transfer to my pc. From my PC there is also a full image to an external drive (using Macrium Reflect, a free for home AND business diamond-software). So i'm covered and many times i restored a full site (after a bad Wordpress update). I suggest your customers do the very same.

 

Anyway, no reason to feel bad and apologize to us so much, just be sure you spend all your time restoring the sites. Tickets can get an automated reply.

 

Good luck !!!!


  • 0

#45 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 07:50 AM

This backup system is new and was to replace the old system you had issues with. It has worked well and to be honest it's not the software that is the issue this time, but the hardware powering it. The old backup system - it was all software issues.

 

I am aware that if this current backup were to be damaged we'd have nothing to restore.  It's not something that hasn't occurred to me but it was not intended to be the primary form of data protection.

 

I do also suggest our customers - and any customer of any hosting provider everywhere - take their own periodic backups.  If any clients have a cPanel backup of their account they want restored - we can restore that and get them back online extremely quickly.

 

As it stands right now - I'm just watching a copy bar progress - there's nothing at this moment I can do to assist with or speed up the process of restoring services or I would be doing just that.

 

I will be posting status updates here on this very thread.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#46 Gregory Wells

Gregory Wells

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 23 September 2018 - 09:24 AM

Ditto for me in all of the previous poster's comments.

 

I'll be a customer as long as I have a need to be so.


  • 0

#47 bobptz

bobptz

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 23 September 2018 - 09:46 AM

I am not an admin, but here are some thoughts.  Please don;t laugh if they are techically impossible.

 

I initially thought it might be a good idea to physically transfer the backup server from Phoenix to Denver.  Then I saw that they are 943 km flying distance, or 1321.36 km by car. 

 

What if you had created servers in Phoenix, where the backup server is located and restore to them with realy high speed?   Would this be possible?


  • 0

#48 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 09:52 AM

I am not an admin, but here are some thoughts.  Please don;t laugh if they are techically impossible.

 

I initially thought it might be a good idea to physically transfer the backup server from Phoenix to Denver.  Then I saw that they are 943 km flying distance, or 1321.36 km by car. 

 

What if you had created servers in Phoenix, where the backup server is located and restore to them with realy high speed?   Would this be possible?

The issue wasn't that the distance slowed things down.  We can transfer data at the rate of gigabytes per second from Phoenix to Denver.  It's the backup hardware itself that is causing problems.

 

We're copying the data off of this backup server in the fastest way possible and moving that new secondary copy of the data to a server with the capability to handle the data at the speed we need to handle it for maximum restoration speed.

 

This step would have to happen whether or not both servers were in the same location to begin with or not.

 

We're planning for 50% of services to be restored within 24 hours and 100% by 48.  Based upon our estimations it should take a fair bit less than this but we don't want to promise a short time and end up taking longer if we can help it.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#49 Kraken

Kraken

    Newbie

  • Members
  • Pip
  • 13 posts

Posted 23 September 2018 - 11:05 AM

Is there any advantage whatsoever to opening a support ticket? I'm losing money while my store is offline so it would be nice to get into a priority queue, if you have one. I think I'm on server NS1, but my info might be outdated.

 

I don't want to make extra work for anybody if a ticket would be irrelevant at this point. I appreciate your open communication, I've been a satisfied customer for many years, and I won't hold this against you. But I sure would like to get my store back ASAP, and I'll need to know how old the database is.


  • 0

#50 Laimonas

Laimonas

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male
  • Location:European Union
  • Interests:wine

Posted 23 September 2018 - 11:10 AM

I am on S1 and my site is still down.
  • 0

#51 KevinD872

KevinD872

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:NW Indiana
  • Interests:Drums & music, motorcycles, computers, programming and gadgets.

Posted 23 September 2018 - 11:45 AM

I've been with MDD for a little over 10 years, and I can definitely vouch for their normally rock-solid service, and they definitely provide excellent support under normal circumstances.  With this said, I have no intentions of leaving MDD and if anything I think I will actually have even more confidence after the dust settles since I have no doubt that they will do whatever they can to prevent such a catastrophe in the future.

 

I'm also glad to hear that MDD still has no plans to sell the business.  I remember specifically asking about this when I first signed up because it is true that this almost always results in the provider's quality of service taking a nosedive.  I've been burned before, and I know this.  I hope and trust that should your son(s) indeed take over one day that they will have as much integrity and pride in their service as you do.

 

Regarding the less than desirable message that appears when customers attempt to visit our websites:

Rather than having us clients create yet another ticket just so we can replace it with a better message or temporary website, would it be feasible to just do this for us all on our behalf?  I'm sure I'm not alone in thinking that a message stating "The host provider for this website has suffered a catastrophic failure and we are working around the clock to restore service ASAP.  We sincerely apologize for the inconvenience." would be better than it is now as it looks more like the website was abandoned or left misconfigured for days.  I don't know what all would be involved in order to do this for all clients, but on the surface anyway, it would only be one single and small HTML "index" file to copy to each domain's public folder.


  • 1

#52 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 23 September 2018 - 12:03 PM

As it stands right now - I'm just watching a copy bar progress - there's nothing at this moment I can do to assist with or speed up the process of restoring services or I would be doing just that.
 
I will be posting status updates here on this very thread.

 
 

Regarding the less than desirable message that appears when customers attempt to visit our websites:
Rather than having us clients create yet another ticket just so we can replace it with a better message or temporary website, would it be feasible to just do this for us all on our behalf?  I'm sure I'm not alone in thinking that a message stating "The host provider for this website has suffered a catastrophic failure and we are working around the clock to restore service ASAP.  We sincerely apologize for the inconvenience." would be better than it is now as it looks more like the website was abandoned or left misconfigured for days.  I don't know what all would be involved in order to do this for all clients, but on the surface anyway, it would only be one single and small HTML "index" file to copy to each domain's public folder.

 
Very much agree.  It makes more sense to go ahead and implement this servers wide as a gesture of good will and to preserve our reputations. 
 
In the meantime I've sent a ticket asking for copies so I can consider alternatives given that I am on server 5 and  Server 1 is only just completed.
  • 0

#53 cziv

cziv

    Member

  • Members
  • PipPip
  • 52 posts
  • Gender:Male

Posted 23 September 2018 - 12:16 PM

Is the restoration of S1 going well ?


  • 0

#54 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 12:23 PM

I've been with MDD for a little over 10 years, and I can definitely vouch for their normally rock-solid service, and they definitely provide excellent support under normal circumstances.  With this said, I have no intentions of leaving MDD and if anything I think I will actually have even more confidence after the dust settles since I have no doubt that they will do whatever they can to prevent such a catastrophe in the future.

 

I'm also glad to hear that MDD still has no plans to sell the business.  I remember specifically asking about this when I first signed up because it is true that this almost always results in the provider's quality of service taking a nosedive.  I've been burned before, and I know this.  I hope and trust that should your son(s) indeed take over one day that they will have as much integrity and pride in their service as you do.

 

Regarding the less than desirable message that appears when customers attempt to visit our websites:

Rather than having us clients create yet another ticket just so we can replace it with a better message or temporary website, would it be feasible to just do this for us all on our behalf?  I'm sure I'm not alone in thinking that a message stating "The host provider for this website has suffered a catastrophic failure and we are working around the clock to restore service ASAP.  We sincerely apologize for the inconvenience." would be better than it is now as it looks more like the website was abandoned or left misconfigured for days.  I don't know what all would be involved in order to do this for all clients, but on the surface anyway, it would only be one single and small HTML "index" file to copy to each domain's public folder.

 
 
 
Very much agree.  It makes more sense to go ahead and implement this servers wide as a gesture of good will and to preserve our reputations. 
 
In the meantime I've sent a ticket asking for copies so I can consider alternatives given that I am on server 5 and  Server 1 is only just completed.

I have been trying to think of some way that we could automate this.  For each account create we have to kill the DNS Zone of the primary domain on the account and then we have to create the account.  Then we have to notify the client that we've done this as well as the new account details.  Beyond that we'll have to overwrite the accounts with our backups without overwriting anybody that's already been restored.  Additionally we'll have to be sure not to overwrite any accounts where the account was set up blank but the client decided to restore data and recover on their own.

 

If I can come up with a good way to do it that won't divert resources away from the restoration process we'll do it.  I can't think of any way to make it happen that doesn't involve communicating with us in some way, shape, or form.

 

Is the restoration of S1 going well ?

We're presently waiting for the facility to move the disks from the old backup server to the new one.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#55 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 12:24 PM

Regarding being on S5 - we are really hoping to have everything restored within 48 hours with most clients being restored within 24.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#56 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 23 September 2018 - 12:28 PM

I have been trying to think of some way that we could automate this.  For each account create we have to kill the DNS Zone of the primary domain on the account and then we have to create the account.  Then we have to notify the client that we've done this as well as the new account details.  Beyond that we'll have to overwrite the accounts with our backups without overwriting anybody that's already been restored.  Additionally we'll have to be sure not to overwrite any accounts where the account was set up blank but the client decided to restore data and recover on their own.

 

If I can come up with a good way to do it that won't divert resources away from the restoration process we'll do it.  I can't think of any way to make it happen that doesn't involve communicating with us in some way, shape, or form.

 

We're presently waiting for the facility to move the disks from the old backup server to the new one.

 

 

Can you edit the "cgi-sys/defaultwebpage.cgi" file?  Since it is already there, there's no need to kill/create/overwrite anything.


  • 0

#57 Devem

Devem

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 23 September 2018 - 12:37 PM

Michael,

 

Have you just implemented an automated message for tickets? It is now sounding spammy and my ticket has had 6 automated replies in a row.

 

I rather you do not implement this automated response.

 

We are sorry for not being able to respond to support tickets as quickly as we normally do. We are doing our best to answer everybody in detail.

All tickets are presently being answered in the order they are received.


  • 0

#58 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 12:51 PM

 

 

Can you edit the "cgi-sys/defaultwebpage.cgi" file?  Since it is already there, there's no need to kill/create/overwrite anything.

Already working on that.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#59 Brad

Brad

    Member

  • Members
  • PipPip
  • 29 posts

Posted 23 September 2018 - 12:52 PM

Great!
cPanel Web Template Editor - defaultwebpage.cgi
  • 0

#60 KevinD872

KevinD872

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:NW Indiana
  • Interests:Drums & music, motorcycles, computers, programming and gadgets.

Posted 23 September 2018 - 12:56 PM

Already working on that.

Good to hear.  It sounds like this is likely the best way to go.  (Thanks for the idea, Brad!)

 

If you do this, just be sure to change it back to the original default once this is all said and done.  Otherwise it will just be confusing when the real reason is indeed a configuration problem.   :D


  • 0




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users