webeye Posted September 23, 2018 Report Share Posted September 23, 2018 I understand the pecking order at this time is S1, followed by P1, and then R1. How many servers are there? I'm on R3 - would it be last? Is the recovery time objective still up to 96 hours - 4 days? Quote Link to comment Share on other sites More sharing options...
jwinter Posted September 23, 2018 Report Share Posted September 23, 2018 my sites on P1 are running and have been since yesterday (Saturday) mid-afternoon. Also sites on P2 are running, they were up before P1 sites were back. are you going to restore over what is already there? I have confirmed emails are working currently. would another restore wipe out emails received at the server since it went back online? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 MDD, do you have any idea when R3 reseller servers will be restored. I see different orders that you are posting but not this one. Some of my websites are backup on this server but many are not. DOes that mean you are still loading those?We are restoring what we can as we can - this won't be an 'everybody is back online at the same time' type of thing - although I wish that were possible. I understand the pecking order at this time is S1, followed by P1, and then R1. How many servers are there? I'm on R3 - would it be last? Is the recovery time objective still up to 96 hours - 4 days?The current plan is S1, P1, R1. After that we are likely going to go S2, P2, R2. It is worth noting that S2, 3, 4, and 5 as well as P2, R2, R3, and R4 all hold far less data than the *1 servers - meaning each of those should go faster. We're really hoping for no more than 48 hours - ideally 24 hours for most accounts to be online but we won't be able to provide a real ETA until we begin restoring backups from the new backup server to the client servers. That's the only thing that's really potentially variable at this point. mine sites on P1 are running and have been since yesterday (Saturday) mid-afternoon. Also sites on P2 are running, they were up before P1 sites were back. are you going to restore over what is already there? I have confirmed emails are working currently. would another restore wipe out emails received at the server since it went back online?We won't be restoring over any already-restored accounts. Quote Link to comment Share on other sites More sharing options...
giantjack Posted September 23, 2018 Report Share Posted September 23, 2018 Mike can you please confirm why some websites are redirecting to others on your network. E.g: https://whmcsportal.com/ Once the SSL is accepted it'll redirect to another site unrelated to me Quote Link to comment Share on other sites More sharing options...
Tandemia Posted September 23, 2018 Report Share Posted September 23, 2018 @MikeDVB Could you please clarify the status of the restoration of P1. I am confused about jwinter reporting that his websites on P1 are okay while you saying that P1 backup image is being prepared for copy. Thank you for your time. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 Mike can you please confirm why some websites are redirecting to others on your network. E.g: https://whmcsportal.com/ Once the SSL is accepted it'll redirect to another site unrelated to me This will happen if you don't have SSL installed on your own site and you're on an IP with another site that has SSL. If you want a blank account to put up something of your own / install SSL for free via Let's Encrypt let us know in a ticket. If you don't - just let us know your domain and we can kill the DNS zone if you're using our DNS which will cause it not to resolve [and therefor not show anything at all]. @MikeDVB Could you please clarify the status of the restoration of P1. I am confused about jwinter reporting that his websites on P1 are okay while you saying that P1 backup image is being prepared for copy. Thank you for your time.There are a lot of customers on P1 that were restored from the backup server directly via the slow method. You are correct in that we are copying the backup of P1 from the backup server to the new restoration server. We have not yet started restoring it using the fast method and will do so as soon as it's possible to do so. Here in a few hours - and then it shouldn't take more than a few hours to restore everything ideally after that - but we're waiting until we start doing actual direct restores to gauge the speed and give a real ETA. Quote Link to comment Share on other sites More sharing options...
Centextalk.com Posted September 23, 2018 Report Share Posted September 23, 2018 48 hours and we still know nothing. Have any sites been restored today? Quote Link to comment Share on other sites More sharing options...
cziv Posted September 23, 2018 Report Share Posted September 23, 2018 It seems that for me the message is now (at the bottom) "Status: 403 Forbidden Content-Type: text/plain; charset=utf-8 403 Forbidden Executing in an invalid environment for the supplied user" at S3. S2 seems to display the new message. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 48 hours and we still know nothing. Have any sites been restored today?I've provided a lot of information, what exactly would you like to know? About 20% of total sites have been restored using the slow method, we expect to restore the remaining 80% over the next 48 hours. Quote Link to comment Share on other sites More sharing options...
ki4csq@gmail.com Posted September 23, 2018 Report Share Posted September 23, 2018 R3? Can we use the Dave Ramsey approach and knock out the smallest first? |I am at risk of losing my two largest clients, they are already demanding their email and files, which I don't have and can't get. I lose either of them, I am out of business. Quote Link to comment Share on other sites More sharing options...
Avatar Posted September 23, 2018 Report Share Posted September 23, 2018 Our server is p1. Our domain is question is avatarrelo.com. This is URGENT: Our twenty-seven (27) mailing lists are gone; they do not appear in control panel. These mailing lists represent a significant portion of the email which enters our organization and the proper functioning of these lists is critical to us. Please restore our mail lists. We need them back up and functioning as soon as possible. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 R3? Can we use the Dave Ramsey approach and knock out the smallest first? | I am at risk of losing my two largest clients, they are already demanding their email and files, which I don't have and can't get. I lose either of them, I am out of business. The smallest server would actually be P2 followed by R4. You aren't the only one in that situation as we have many clients in that situation. I do wish there was something I could do to get everybody online that stands to lose money / clients / sales before those that don't - but unfortunately we're working on the whole-server level and not the individual-account level. Our server is p1. Our domain is question is avatarrelo.com. This is URGENT: Our twenty-seven (27) mailing lists are gone; they do not appear in control panel. These mailing lists represent a significant portion of the email which enters our organization and the proper functioning of these lists is critical to us. Please restore our mail lists. We need them back up and functioning as soon as possible.Are you referring to MailMan? If so, do please open a ticket and we'll get with you as quickly as possible. Quote Link to comment Share on other sites More sharing options...
jsolo Posted September 23, 2018 Report Share Posted September 23, 2018 @Mike thanks for validating. The validation email was never received anywhere (gmail, domain registered with you, etc). I was able to get my account partially restored manually by having you create a 'new' account. I then downloaded the backup file from domain cpanel\full account backups\generate download. This produced a rather large (2.5gb) .gz file which I then restored through ssh to the home directory. Files in the homedir had to be relocated, but mostly everything else appears to be in the right place. What's missing are the dns records, subdomains, databases and some others. Jetbackup documentation references a restore button that should be present next to the generate backup button. However it's not there. How can I fully restore the account myself, including the above missing items? My concern is if the automated restore runs, new emails which have been received since the crash will be wiped from the mail storage. Some have been downloaded, but some clients also use imap. Thanks! Quote Link to comment Share on other sites More sharing options...
Centextalk.com Posted September 23, 2018 Report Share Posted September 23, 2018 I've provided a lot of information, what exactly would you like to know? About 20% of total sites have been restored using the slow method, we expect to restore the remaining 80% over the next 48 hours.My question was petty straight forward. We've been told a lot, such as, we could put up a page, but that has not come to fruition. Quote Link to comment Share on other sites More sharing options...
AMC4x4 Posted September 23, 2018 Report Share Posted September 23, 2018 Mike can you please confirm why some websites are redirecting to others on your network. E.g: https://whmcsportal.com/ Once the SSL is accepted it'll redirect to another site unrelated to me Yeah, iTunes scrapes my blog and posts a podcast and they ended up removing it, I guess because the SSL version of the feed URL seems to point to some site about Emiliya. Maybe they thought I had been hacked. I moved that blog over to a new hosting account and I hope I haven't lost all my reviews and ratings. The blog/podcast doesn't get a lot of traffic, but I worked hard to earn what little I have. Quote Link to comment Share on other sites More sharing options...
AMC4x4 Posted September 23, 2018 Report Share Posted September 23, 2018 @Mike thanks for validating. The validation email was never received anywhere (gmail, domain registered with you, etc). I was able to get my account partially restored manually by having you create a 'new' account. I then downloaded the backup file from domain cpanel\full account backups\generate download. This produced a rather large (2.5gb) .gz file which I then restored through ssh to the home directory. Files in the homedir had to be relocated, but mostly everything else appears to be in the right place. What's missing are the dns records, subdomains, databases and some others. Jetbackup documentation references a restore button that should be present next to the generate backup button. However it's not there. How can I fully restore the account myself, including the above missing items? My concern is if the automated restore runs, new emails which have been received since the crash will be wiped from the mail storage. Some have been downloaded, but some clients also use imap. Thanks! Mike, I'm wondering if part of the sudden slowdowns you might have experienced from the backup server were a result of some of us grabbing full account backups while you were also trying to restore? I did this before you mentioned any speed issues on your part, but the backup did take a long time. It looks like others had the same idea as soon as they regained access to your account. You said you had little vision into what the backup server was doing, but I would think this would have been part of the issue - you'd be humming along, and then probably one of us started grabbing a backup, causing a slowdown for you. I wonder if there's a way to turn off client generated backups until you get the data off that machine? Just a thought if you hadn't thought of it already. Quote Link to comment Share on other sites More sharing options...
glennk Posted September 23, 2018 Report Share Posted September 23, 2018 Mike, I'm wondering if part of the sudden slowdowns you might have experienced from the backup server were a result of some of us grabbing full account backups while you were also trying to restore? I did this before you mentioned any speed issues on your part, but the backup did take a long time. It looks like others had the same idea as soon as they regained access to your account. You said you had little vision into what the backup server was doing, but I would think this would have been part of the issue - you'd be humming along, and then probably one of us started grabbing a backup, causing a slowdown for you. I wonder if there's a way to turn off client generated backups until you get the data off that machine? Just a thought if you hadn't thought of it already.Probably not because not many people can get in. Quote Link to comment Share on other sites More sharing options...
cziv Posted September 23, 2018 Report Share Posted September 23, 2018 I think there is a very long road ahead of us all. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 @Mike thanks for validating. The validation email was never received anywhere (gmail, domain registered with you, etc). I was able to get my account partially restored manually by having you create a 'new' account. I then downloaded the backup file from domain cpanel\full account backups\generate download. This produced a rather large (2.5gb) .gz file which I then restored through ssh to the home directory. Files in the homedir had to be relocated, but mostly everything else appears to be in the right place. What's missing are the dns records, subdomains, databases and some others. Jetbackup documentation references a restore button that should be present next to the generate backup button. However it's not there. How can I fully restore the account myself, including the above missing items? My concern is if the automated restore runs, new emails which have been received since the crash will be wiped from the mail storage. Some have been downloaded, but some clients also use imap. Thanks!You would let us know the full backup name and we can restore your entire account from the backup just like it was when you took the backup. You won't be able to restore from our JetBackup server right now. We can not restore over your accounts and then when you get the email that the server restoration is complete you can let us know if you want us to restore over them at that point or do anything special if you like. My question was petty straight forward. We've been told a lot, such as, we could put up a page, but that has not come to fruition.I'm not sure I follow again, and I'm sorry. If you want a blank account to put up a temporary page - you need a support ticket. If you have one and it hasn't been answered yet - I'm sorry - we're answering them as fast as we can. Yeah, iTunes scrapes my blog and posts a podcast and they ended up removing it, I guess because the SSL version of the feed URL seems to point to some site about Emiliya. Maybe they thought I had been hacked. I moved that blog over to a new hosting account and I hope I haven't lost all my reviews and ratings. The blog/podcast doesn't get a lot of traffic, but I worked hard to earn what little I have. I understand and I am sorry, I wish there was something I could do now to have prevented this. We screwed up, no way around it. Mike, I'm wondering if part of the sudden slowdowns you might have experienced from the backup server were a result of some of us grabbing full account backups while you were also trying to restore? I did this before you mentioned any speed issues on your part, but the backup did take a long time. It looks like others had the same idea as soon as they regained access to your account. You said you had little vision into what the backup server was doing, but I would think this would have been part of the issue - you'd be humming along, and then probably one of us started grabbing a backup, causing a slowdown for you. I wonder if there's a way to turn off client generated backups until you get the data off that machine? Just a thought if you hadn't thought of it already. We can turn off client access to the backup system but it hasn't been an issue at all thus far. I think there is a very long road ahead of us all. This new restoration process should shorten that road. We're about to begin the account restorations for S1 now. P1 is still copying and is about 70% done. Quote Link to comment Share on other sites More sharing options...
Centextalk.com Posted September 23, 2018 Report Share Posted September 23, 2018 I'm not sure I follow again, and I'm sorry. If you want a blank account to put up a temporary page - you need a support ticket. If you have one and it hasn't been answered yet - I'm sorry - we're answering them as fast as we can. The question was if any sites had been restored today. It's been seven hours since I submitted the ticket to be able to put a page up. Quote Link to comment Share on other sites More sharing options...
KevinJones Posted September 23, 2018 Report Share Posted September 23, 2018 The default page should be better now for when your site doesn't load should be better now.Mines still looks the same from here. "404 Not Found" "The server can not find the requested page". To the user it would like like my site went away for some unknown reason. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 The question was if any sites had been restored today. It's been seven hours since I submitted the ticket to be able to put a page up. No, the only thing we've been able to do due to the issues with the old backup server are copying those server backups off onto better hardware. Now that those are finishing we can start restoring back to cPanel accounts. We're actually working on doing that right now for S1. Once a few accounts are restored we'll be able to give a better ETA. Mines still looks the same from here. "404 Not Found" "The server can not find the requested page". To the user it would like like my site went away for some unknown reason.You can have us create a blank account for you so you can put up your own message if you like, just reach out. Quote Link to comment Share on other sites More sharing options...
jsolo Posted September 23, 2018 Report Share Posted September 23, 2018 You would let us know the full backup name and we can restore your entire account from the backup just like it was when you took the backup. You won't be able to restore from our JetBackup server right now. We can not restore over your accounts and then when you get the email that the server restoration is complete you can let us know if you want us to restore over them at that point or do anything special if you like. I'm not sure I understand what this means. If you restore now, the automated restore will skip my account then? That's fine. I've updated ticket #657811 with the path/file name. Thank you. Quote Link to comment Share on other sites More sharing options...
ki4csq@gmail.com Posted September 23, 2018 Report Share Posted September 23, 2018 Once you guys get us back up and going, I need you to install a backup to Dropbox option on the WHM and Cpanel. Thank you. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Report Share Posted September 23, 2018 I'm not sure I understand what this means. If you restore now, the automated restore will skip my account then? That's fine. I've updated ticket #657811 with the path/file name. Thank you. If we restore your backup - it's up to you if you want us to overwrite it or not. Once you guys get us back up and going, I need you to install a backup to Dropbox option on the WHM and Cpanel. Thank you. I wasn't aware that was an option, will have to look into it. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.