Michael D. Posted September 23, 2018 Author Report Share Posted September 23, 2018 The S1 backup image is still copying to the 24 Drive SSD array for restoration. The P1 backup image is being created on the 4 Disk SSD array in preparation for copying to the 24 Drive system. 1 Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Author Report Share Posted September 23, 2018 We are now restoring backups for the S1 server - this new backup server is going EXCEPTIONALLY fast so far in comparison to restorations from the old backup server. Once we have gotten more accounts restored we'll give our best ETA we can come up with on S1. We are still copying the P1 backup at this time and will be starting on the R1 backup shortly. Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Author Report Share Posted September 23, 2018 S1 restorations are in the accounts starting with BN. Link to comment Share on other sites More sharing options...
Michael D. Posted September 23, 2018 Author Report Share Posted September 23, 2018 The restorations from the new backup system to the S1 server are going about 1000% faster than the old backup server by my estimations. I'm still trying to work out a good ETA - and I need to wait on some more accounts to restore before I have a data set big enough to make such an estimation. 2 Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 We are restoring accounts at about the rate of 900 per hour from the new backup server and have approximately 9,000 accounts to restore. Assuming everything scales smoothly we should be able to have everything restored within 10 to 20 hours from this point. This is a very tentative ETA. It is possible it may still take up to 48 hours for 100% as it will still take time to copy the full backups off of the old backup server and onto the new one and due to the nature of the old backup server we can only do one at a time. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 Here are the current calculated ETA's: p1 Monday, September 24, 2018 at 1:00:00 AMr1 Monday, September 24, 2018 at 10:00:00 AMs2 Monday, September 24, 2018 at 8:00:00 PMp2 Tuesday, September 25, 2018 at 1:00:00 AMr2 Tuesday, September 25, 2018 at 10:00:00 AMs3 Tuesday, September 25, 2018 at 10:00:00 PMr3 Wednesday, September 26, 2018 at 6:00:00 AMs4 Wednesday, September 26, 2018 at 4:00:00 PMr4 Wednesday, September 26, 2018 at 10:00:00 PMs5 Thursday, September 27, 2018 at 7:00:00 AMs0 Thursday, September 27, 2018 at 1:00:00 PM These are likely over-estimations [i.e. should be restored before this] and we feel that this is likely worst case. The reason we won't be done in as long as it would simply take to restore the data to the server is because we're strapped down by the transfer rate between the old backup server and the new one with it taking about 8 hours to transfer ~3TB of data. These times are for 100% restoration of all services. We will update this thread if this changes. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 Latest ETA Calculations: s1 Sunday, September 23, 2018 at 11:59:00 PMp1 Monday, September 24, 2018 at 4:00:00 AMr1 Monday, September 24, 2018 at 4:00:00 PMs2 Monday, September 24, 2018 at 4:00:00 PMp2 Monday, September 24, 2018 at 3:00:00 PMr2 Tuesday, September 25, 2018 at 2:00:00 AMs3 Tuesday, September 25, 2018 at 6:00:00 AMr3 Tuesday, September 25, 2018 at 2:00:00 PMs4 Tuesday, September 25, 2018 at 5:00:00 PMr4 Tuesday, September 25, 2018 at 6:00:00 PMs5 Tuesday, September 25, 2018 at 10:00:00 PMs0 Tuesday, September 25, 2018 at 10:00:00 PM Link to comment Share on other sites More sharing options...
ericr Posted September 24, 2018 Report Share Posted September 24, 2018 We had to remove the ETA as there was a mistake made on the sheet causing the ETA to be off by a few years(we do not even remotely plan on it taking till march 2019). I do apologize. As for S2 it is still 4th. s1 Monday, September 24, 2018 at 2:00:00 AMp1 Monday, September 24, 2018 at 4:00:00 AMr1 Monday, September 24, 2018 at 6:00:00 PMs2 Monday, September 24, 2018 at 7:00:00 PMp2 Monday, September 24, 2018 at 6:00:00 PMr2 Tuesday, September 25, 2018 at 5:00:00 AMs3 Tuesday, September 25, 2018 at 12:00:00 PMr3 Tuesday, September 25, 2018 at 8:00:00 PMs4 Wednesday, September 26, 2018 at 1:00:00 AMr4 Wednesday, September 26, 2018 at 2:00:00 AMs5 Wednesday, September 26, 2018 at 8:00:00 AMs0 Wednesday, September 26, 2018 at 8:00:00 AM Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 Here are the latest ETA's. The transfers from the old backup server to the new one are taking a little longer than expected. All times are Eastern Daylight Time. s1 Monday, September 24, 2018 at 2:00:00 AM p1 Monday, September 24, 2018 at 4:00:00 AM r1 Monday, September 24, 2018 at 6:00:00 PM s2 Monday, September 24, 2018 at 7:00:00 PM p2 Monday, September 24, 2018 at 6:00:00 PM r2 Tuesday, September 25, 2018 at 5:00:00 AM s3 Tuesday, September 25, 2018 at 12:00:00 PM r3 Tuesday, September 25, 2018 at 8:00:00 PM s4 Wednesday, September 26, 2018 at 1:00:00 AM r4 Wednesday, September 26, 2018 at 2:00:00 AM s5 Wednesday, September 26, 2018 at 8:00:00 AM s0 Wednesday, September 26, 2018 at 8:00:00 AM Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 The R1 server is now being copied from Slow Backup to Fast Backup.The Restoration of accounts from the P1 backups on Fast Backup are starting now. This is fast backup: It is connected via 10GBPS Networking and is running 24 x 1.6TB SSDs via Hardware Raid. The four loose drives in the top are 4 out of 8 SSDs we are using to transfer data from Slow Backup to Fast Backup. This is a temporary solution to get everybody restored as quickly as possible. Our provider in Phoenix, IOFlood, is helping us out with this recovery by letting us use this hardware to speed up the process. They're also handling all of the disk swaps to transfer data from the Slow Backup to the Fast Backup. We couldn't be doing this without them. Here is fast backup hard at work restoring client data: Here's Slow Backup Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 I am going to get a little bit of sleep - Eric will keep updates posted here on the forums and we're still working on support tickets as quickly as we can. Link to comment Share on other sites More sharing options...
ericr Posted September 24, 2018 Report Share Posted September 24, 2018 Status update:All accounts on S1 are restored. The p1 archive is 24% of the way of being restored onto the 10G ssd server.The r1 backup is about halfway done being extracted from the backup server in preparation to being restored on the 10G ssd server. Link to comment Share on other sites More sharing options...
ericr Posted September 24, 2018 Report Share Posted September 24, 2018 We were doing restores differently initially and that process was significantly slower. We are not overwriting any accounts unless specifically told to. Link to comment Share on other sites More sharing options...
ericr Posted September 24, 2018 Report Share Posted September 24, 2018 I have started the restoration of p1's accounts at this time. Link to comment Share on other sites More sharing options...
ericr Posted September 24, 2018 Report Share Posted September 24, 2018 With the exception of one single account P1 is done with it's restorations. The account that we are waiting on is on the larger side so I did not want to save the status update on it. Link to comment Share on other sites More sharing options...
ericr Posted September 24, 2018 Report Share Posted September 24, 2018 Current ETA based on the progress up till now(Eastern Time):s1 Completedp1 Completedr1 Monday, September 24, 2018 at 11:00:00 PMp2 Monday, September 24, 2018 at 3:00:00 PMs2 Tuesday, September 25, 2018 at 1:00:00 AMr2 Tuesday, September 25, 2018 at 9:00:00 AMs3 Tuesday, September 25, 2018 at 4:00:00 PMr3 Wednesday, September 26, 2018 at 12:00:00 AMs4 Wednesday, September 26, 2018 at 4:00:00 AMr4 Wednesday, September 26, 2018 at 6:00:00 AMs5 Wednesday, September 26, 2018 at 12:00:00 PMs0 Wednesday, September 26, 2018 at 12:00:00 PM Link to comment Share on other sites More sharing options...
richardmichael Posted September 24, 2018 Report Share Posted September 24, 2018 I have some general questions that anyone may answer, as I do not want to take up any of MDD's time.I am not well versed in any of this hosting stuff. I am a mechanical engineer whose company website and e-mail server are hosted by MDD. I am one of the two employees, total, and will be the one to handle any of the work for this. 1. How do I find out what server I am on?2. Is it safe to assume that being a newer client (January 2018), my server will be one of the last restored?3. Will I have to do anything once things are back up and running? Thanks in advance.1. You should be able to find it on the MDDHosting website, under your services. Go here https://www.mddhosting.com/support/clientarea.php?action=services, and for each service listed, click on it, then scroll down a bit and you will see something similar: https://www.dropbox.com/s/ar7g0ghrwjhj8kq/servers.PNG?dl=0You can see that my service is hosted on R4. 2. Probably likely 3. If you are using 3rd party DNS, you have a chance of needing to update your IP address configuration. If you don't use 3rd party DNS, you will want to still monitor anyway and possibly open a ticket after restoration if you get an error about the IP address having changed. Edit: After re-reading I see you may not be familiar about your DNS situation. You can visit https://intodns.com and plug in your domain name(s) (one at a time, though), and run the report. the very first section of the resulting report will tell you the Nameservers being used. If you see supportedns like in my screenshot, you are using MDDHosting's DNS and are less likely to have an IP Address issues after the restoration reaches your server. 2 Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 Here are revised ETA's: s1 Completedp1 Completedr1 Tuesday, September 25, 2018 at 3:00:00 AM EDTp2 Monday, September 24, 2018 at 5:00:00 PM EDTs2 Tuesday, September 25, 2018 at 5:00:00 AM EDTr2 Tuesday, September 25, 2018 at 1:00:00 PM EDTs3 Tuesday, September 25, 2018 at 8:00:00 PM EDTr3 Wednesday, September 26, 2018 at 4:00:00 AM EDTs4 Wednesday, September 26, 2018 at 8:00:00 AM EDTr4 Wednesday, September 26, 2018 at 9:00:00 AM EDTs5 Wednesday, September 26, 2018 at 4:00:00 PM EDTs0 Wednesday, September 26, 2018 at 4:00:00 PM EDT While I do apologize they keep changing - they are estimates based off of our best working information at the time. This is one of the reasons I'm not a huge fan of ETA's - they create expectations that maybe aren't kept up with. This is one of the reasons we keep updating them as we can to remain as accurate as possible. We are presently copying the backups for R1 and P2 from the intermediary SSDs in FastBackup to the restoration array and the S2 backup from SlowBackup to a set of intermediary SSDs. We will begin restoring R1 and P2 as soon as possible and it looks like P2 will begin in about an hour. R1 looks to be taking longer than expected with an estimated start time of about 3 hours from now. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 I am working quickly to separate out client discussions in this thread into the new thread for this purpose -> https://forums.mddhosting.com/topic/1583-major-outage-092118-client-discussion/ 1 Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 I had an email sent to all account holders with an active account on S1 and P2 that outlines DNS / IP Changes / Signing into cPanel and finding the new IP. We're doing our best to keep up with support tickets, however, we are swamped and an hour or so behind on responses right now. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 We will be starting the restorations of the P2 server in approximately 50 minutes and the R1 server in approximately 3 hours and 20 minutes. These are simply the ETA's provided by the data copy processes inside of FastBackup to move the data to the array for restoration. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 The restoration of accounts on the P2 server has started. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 Current ETA to restoration on R1 is 2 hours 45 minutes. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 The P2 server is almost completely restored. Not many accounts left to go. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 If you are not online yet - make sure to try accessing your cPanel from our client area. If you can log into your cPanel your account is already restored and your domain is most likely not pointing to the right IP. If your account isn't restored yet and you have backups of your own you want us to restore, please open a support ticket. Link to comment Share on other sites More sharing options...
Recommended Posts