Jump to content


Photo

Major Outage - 09/21/18+ - Client Discussion


  • Please log in to reply
419 replies to this topic

#81 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 04:00 PM

Mike can you please confirm why some websites are redirecting to others on your network. E.g:

 

https://whmcsportal.com/

 

Once the SSL is accepted it'll redirect to another site unrelated to me

 

This will happen if you don't have SSL installed on your own site and you're on an IP with another site that has SSL.  If you want a blank account to put up something of your own / install SSL for free via Let's Encrypt let us know in a ticket.  If you don't - just let us know your domain and we can kill the DNS zone if you're using our DNS which will cause it not to resolve [and therefor not show anything at all].

 

@MikeDVB Could you please clarify the status of the restoration of P1.

 

I am confused about jwinter reporting that his websites on P1 are okay while you saying that P1 backup image is being prepared for copy.

 

Thank you for your time.

There are a lot of customers on P1 that were restored from the backup server directly via the slow method.  You are correct in that we are copying the backup of P1 from the backup server to the new restoration server.  We have not yet started restoring it using the fast method and will do so as soon as it's possible to do so.  Here in a few hours - and then it shouldn't take more than a few hours to restore everything ideally after that - but we're waiting until we start doing actual direct restores to gauge the speed and give a real ETA.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#82 Centextalk.com

Centextalk.com

    Newbie

  • Members
  • Pip
  • 3 posts
  • Gender:Male

Posted 23 September 2018 - 04:09 PM

48 hours and we still know nothing.

Have any sites been restored today?
  • 0

#83 cziv

cziv

    Member

  • Members
  • PipPip
  • 52 posts
  • Gender:Male

Posted 23 September 2018 - 04:15 PM

It seems that for me the message is now (at the bottom) "Status: 403 Forbidden Content-Type: text/plain; charset=utf-8 403 Forbidden Executing in an invalid environment for the supplied user" at S3.

 

S2 seems to display the new message.


  • 0

#84 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 04:15 PM

48 hours and we still know nothing.

Have any sites been restored today?

I've provided a lot of information, what exactly would you like to know?  About 20% of total sites have been restored using the slow method, we expect to restore the remaining 80% over the next 48 hours.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#85 ki4csq@gmail.com

ki4csq@gmail.com

    Faith Tech

  • Members
  • Pip
  • 2 posts
  • Gender:Male

Posted 23 September 2018 - 04:19 PM

R3?
 

Can we use the Dave Ramsey approach and knock out the smallest first? :)|

I am at risk of losing my two largest clients, they are already demanding their email and files, which I don't have and can't get.  I lose either of them, I am out of business. 


  • 0

#86 Avatar

Avatar

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 23 September 2018 - 04:19 PM

Our server is p1. Our domain is question is avatarrelo.com.

 

This is URGENT: Our twenty-seven (27) mailing lists are gone; they do not appear in control panel. These mailing lists represent a significant portion of the email which enters our organization and the proper functioning of these lists is critical to us. Please restore our mail lists. We need them back up and functioning as soon as possible.


  • 0

#87 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 04:24 PM

R3?
 

Can we use the Dave Ramsey approach and knock out the smallest first? :)|

I am at risk of losing my two largest clients, they are already demanding their email and files, which I don't have and can't get.  I lose either of them, I am out of business. 

The smallest server would actually be P2 followed by R4.  You aren't the only one in that situation as we have many clients in that situation.  I do wish there was something I could do to get everybody online that stands to lose money / clients / sales before those that don't - but unfortunately we're working on the whole-server level and not the individual-account level.

 

Our server is p1. Our domain is question is avatarrelo.com.

 

This is URGENT: Our twenty-seven (27) mailing lists are gone; they do not appear in control panel. These mailing lists represent a significant portion of the email which enters our organization and the proper functioning of these lists is critical to us. Please restore our mail lists. We need them back up and functioning as soon as possible.

Are you referring to MailMan?  If so, do please open a ticket and we'll get with you as quickly as possible.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#88 jsolo

jsolo

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male

Posted 23 September 2018 - 04:35 PM

@Mike thanks for validating.  The validation email was never received anywhere (gmail, domain registered with you, etc).

 

I was able to get my account partially restored manually by having you create a 'new' account.

 

I then downloaded the backup file from domain cpanel\full account backups\generate download.  This produced a rather large (2.5gb) .gz file which I then restored through ssh to the home directory.  Files in the homedir had to be relocated, but mostly everything else appears to be in the right place.

 

What's missing are the dns records, subdomains, databases and some others.

 

Jetbackup documentation references a restore button that should be present next to the generate backup button.  However it's not there.  How can I fully restore the account myself, including the above missing items?

 

My concern is if the automated restore runs, new emails which have been received since the crash will be wiped from the mail storage.  Some have been downloaded, but some clients also use imap.

 

Thanks!


  • 0

#89 Centextalk.com

Centextalk.com

    Newbie

  • Members
  • Pip
  • 3 posts
  • Gender:Male

Posted 23 September 2018 - 04:55 PM

I've provided a lot of information, what exactly would you like to know?  About 20% of total sites have been restored using the slow method, we expect to restore the remaining 80% over the next 48 hours.


My question was petty straight forward.

We've been told a lot, such as, we could put up a page, but that has not come to fruition.
  • 0

#90 AMC4x4

AMC4x4

    Newbie

  • Members
  • Pip
  • 24 posts

Posted 23 September 2018 - 04:55 PM

Mike can you please confirm why some websites are redirecting to others on your network. E.g:

 

https://whmcsportal.com/

 

Once the SSL is accepted it'll redirect to another site unrelated to me

 

Yeah, iTunes scrapes my blog and posts a podcast and they ended up removing it, I guess because the SSL version of the feed URL seems to point to some site about Emiliya. Maybe they thought I had been hacked. I moved that blog over to a new hosting account and I hope I haven't lost all my reviews and ratings. The blog/podcast doesn't get a lot of traffic, but I worked hard to earn what little I have. :(


  • 0

#91 AMC4x4

AMC4x4

    Newbie

  • Members
  • Pip
  • 24 posts

Posted 23 September 2018 - 05:11 PM

@Mike thanks for validating.  The validation email was never received anywhere (gmail, domain registered with you, etc).

 

I was able to get my account partially restored manually by having you create a 'new' account.

 

I then downloaded the backup file from domain cpanel\full account backups\generate download.  This produced a rather large (2.5gb) .gz file which I then restored through ssh to the home directory.  Files in the homedir had to be relocated, but mostly everything else appears to be in the right place.

 

What's missing are the dns records, subdomains, databases and some others.

 

Jetbackup documentation references a restore button that should be present next to the generate backup button.  However it's not there.  How can I fully restore the account myself, including the above missing items?

 

My concern is if the automated restore runs, new emails which have been received since the crash will be wiped from the mail storage.  Some have been downloaded, but some clients also use imap.

 

Thanks!

 

Mike, I'm wondering if part of the sudden slowdowns you might have experienced from the backup server were a result of some of us grabbing full account backups while you were also trying to restore? I did this before you mentioned any speed issues on your part, but the backup did take a long time. It looks like others had the same idea as soon as they regained access to your account. You said you had little vision into what the backup server was doing, but I would think this would have been part of the issue - you'd be humming along, and then probably one of us started grabbing a backup, causing a slowdown for you. I wonder if there's a way to turn off client generated backups until you get the data off that machine? Just a thought if you hadn't thought of it already.


  • 0

#92 glennk

glennk

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 23 September 2018 - 05:13 PM

 

Mike, I'm wondering if part of the sudden slowdowns you might have experienced from the backup server were a result of some of us grabbing full account backups while you were also trying to restore? I did this before you mentioned any speed issues on your part, but the backup did take a long time. It looks like others had the same idea as soon as they regained access to your account. You said you had little vision into what the backup server was doing, but I would think this would have been part of the issue - you'd be humming along, and then probably one of us started grabbing a backup, causing a slowdown for you. I wonder if there's a way to turn off client generated backups until you get the data off that machine? Just a thought if you hadn't thought of it already.

Probably not because not many people can get in.


  • 0

#93 cziv

cziv

    Member

  • Members
  • PipPip
  • 52 posts
  • Gender:Male

Posted 23 September 2018 - 05:18 PM

I think there is a very long road ahead of us all.


  • 0

#94 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 05:51 PM

@Mike thanks for validating.  The validation email was never received anywhere (gmail, domain registered with you, etc).

 

I was able to get my account partially restored manually by having you create a 'new' account.

 

I then downloaded the backup file from domain cpanel\full account backups\generate download.  This produced a rather large (2.5gb) .gz file which I then restored through ssh to the home directory.  Files in the homedir had to be relocated, but mostly everything else appears to be in the right place.

 

What's missing are the dns records, subdomains, databases and some others.

 

Jetbackup documentation references a restore button that should be present next to the generate backup button.  However it's not there.  How can I fully restore the account myself, including the above missing items?

 

My concern is if the automated restore runs, new emails which have been received since the crash will be wiped from the mail storage.  Some have been downloaded, but some clients also use imap.

 

Thanks!

You would let us know the full backup name and we can restore your entire account from the backup just like it was when you took the backup.  You won't be able to restore from our JetBackup server right now.  We can not restore over your accounts and then when you get the email that the server restoration is complete you can let us know if you want us to restore over them at that point or do anything special if you like.

 

My question was petty straight forward.

We've been told a lot, such as, we could put up a page, but that has not come to fruition.

I'm not sure I follow again, and I'm sorry.  If you want a blank account to put up a temporary page - you need a support ticket.  If you have one and it hasn't been answered yet - I'm sorry - we're answering them as fast as we can.

 

 

Yeah, iTunes scrapes my blog and posts a podcast and they ended up removing it, I guess because the SSL version of the feed URL seems to point to some site about Emiliya. Maybe they thought I had been hacked. I moved that blog over to a new hosting account and I hope I haven't lost all my reviews and ratings. The blog/podcast doesn't get a lot of traffic, but I worked hard to earn what little I have. :(

I understand and I am sorry, I wish there was something I could do now to have prevented this.  We screwed up, no way around it.

 

 

Mike, I'm wondering if part of the sudden slowdowns you might have experienced from the backup server were a result of some of us grabbing full account backups while you were also trying to restore? I did this before you mentioned any speed issues on your part, but the backup did take a long time. It looks like others had the same idea as soon as they regained access to your account. You said you had little vision into what the backup server was doing, but I would think this would have been part of the issue - you'd be humming along, and then probably one of us started grabbing a backup, causing a slowdown for you. I wonder if there's a way to turn off client generated backups until you get the data off that machine? Just a thought if you hadn't thought of it already.

 

We can turn off client access to the backup system but it hasn't been an issue at all thus far.
 

I think there is a very long road ahead of us all.

 

This new restoration process should shorten that road.

 

We're about to begin the account restorations for S1 now.  P1 is still copying and is about 70% done.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#95 Centextalk.com

Centextalk.com

    Newbie

  • Members
  • Pip
  • 3 posts
  • Gender:Male

Posted 23 September 2018 - 06:03 PM

I'm not sure I follow again, and I'm sorry.  If you want a blank account to put up a temporary page - you need a support ticket.  If you have one and it hasn't been answered yet - I'm sorry - we're answering them as fast as we can.


The question was if any sites had been restored today.

It's been seven hours since I submitted the ticket to be able to put a page up.
  • 0

#96 KevinJones

KevinJones

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 23 September 2018 - 06:13 PM

The default page should be better now for when your site doesn't load should be better now.

Mines still looks the same from here. "404 Not Found" "The server can not find the requested page".  To the user it would like like my site went away for some unknown reason.


  • 0

#97 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 06:17 PM

The question was if any sites had been restored today.

It's been seven hours since I submitted the ticket to be able to put a page up.

 

No, the only thing we've been able to do due to the issues with the old backup server are copying those server backups off onto better hardware.  Now that those are finishing we can start restoring back to cPanel accounts.  We're actually working on doing that right now for S1.  Once a few accounts are restored we'll be able to give a better ETA.

 

Mines still looks the same from here. "404 Not Found" "The server can not find the requested page".  To the user it would like like my site went away for some unknown reason.

You can have us create a blank account for you so you can put up your own message if you like, just reach out.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#98 jsolo

jsolo

    Newbie

  • Members
  • Pip
  • 14 posts
  • Gender:Male

Posted 23 September 2018 - 06:27 PM

You would let us know the full backup name and we can restore your entire account from the backup just like it was when you took the backup.  You won't be able to restore from our JetBackup server right now.  We can not restore over your accounts and then when you get the email that the server restoration is complete you can let us know if you want us to restore over them at that point or do anything special if you like.

 

I'm not sure I understand what this means.  If you restore now, the automated restore will skip my account then?  That's fine.

 

I've updated ticket #657811 with the path/file name.

 

Thank you.


  • 0

#99 ki4csq@gmail.com

ki4csq@gmail.com

    Faith Tech

  • Members
  • Pip
  • 2 posts
  • Gender:Male

Posted 23 September 2018 - 06:31 PM

Once you guys get us back up and going, I need you to install a backup to Dropbox option on the WHM and Cpanel. 
Thank you. 


  • 0

#100 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 06:35 PM

 

I'm not sure I understand what this means.  If you restore now, the automated restore will skip my account then?  That's fine.

 

I've updated ticket #657811 with the path/file name.

 

Thank you.

 

If we restore your backup - it's up to you if you want us to overwrite it or not.

 

Once you guys get us back up and going, I need you to install a backup to Dropbox option on the WHM and Cpanel. 
Thank you. 

I wasn't aware that was an option, will have to look into it.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/




2 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users


    Bing (2)