Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 Some accounts on S1 were restored without MySQL databases - I don't know why this is as of yet as every backup includes database backups - but if you find your databases aren't listed you can restore them via cPanel -> JetBackup -> Database Backups. After doing so you will also need to go to cPanel -> MySQL Databases and re-create your database user and apply it to your databases. We're happy to help you with it if you need it in a ticket and we're doing our best to keep up. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 Time to R1 Restoration is approximately 1 hour 40 minutes. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 The last 8 accounts on P2 are restoring now. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 The copy of data for the S2 server from Slow Backup to Intermediate Transfer DIsks is about 75% completed. Once it's 100% we'll move the intermediate disks over to FastBackup and get it ready for restoration. 1 hour 30 minutes left preparing R1 for restoration from Fast Backup. 2 Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 P2 is pretty much done - the only accounts still restoring are absolutely massive in size. Only two accounts still restoring. Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 P2 server is done. ETA to starting R1 Restorations is ~20 minutes. 2 Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 We are starting the restoration of the R1 server now. 3 Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 We upgraded a storage component on the new Fast Backup server and we are seeing a huge increase in performance over what we were already seeing. This should cut down the total restoration time - we will have updated ETAs once we have conducted some more transfers. 5 Link to comment Share on other sites More sharing options...
Michael D. Posted September 24, 2018 Author Report Share Posted September 24, 2018 We performed some changes on the storage in both backup servers. The fast backup server is now doing 4,770 GB/hour data rates while it was doing 500gb/hour. The old backup server has been doing 318 gb/hour. We're watching to see how much of an improvement the old backup server gets as a result of these changes too. The bottle neck is getting the data from the old backup server to the fast one - but this process is substantially faster than restoring directly from the old backup server which was only doing about 50gb/hour. 5 Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 R1 is restoring now, should be done soon. R4 got finished faster than expected and we should be restoring it soon. S2 is also close to being ready to get restored. 4 Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 We are over 50% restored network-wide. 51.4% actually. We are still hoping to be 100% by tomorrow - but the ETAs are calculations based upon transfer speeds and restoration rates. As you've seen they fluctuate and we're still honestly hoping to be one far sooner than any of the ETAs say. 1 Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 We are restoring R1 still as well as R4. Accounts are actively coming online and will keep doing so until these servers are fully restored. Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 Latest ETAs: s1 Completed p1 Completed r1 In Progress p2 Completed s2 Tuesday, September 25, 2018 at 6:00:00 AM EDTr2 Tuesday, September 25, 2018 at 2:00:00 PM EDTs3 Tuesday, September 25, 2018 at 9:00:00 PM EDTr3 Wednesday, September 26, 2018 at 5:00:00 AM EDTs4 Wednesday, September 26, 2018 at 9:00:00 AM EDTr4 In progress s5 Wednesday, September 26, 2018 at 3:00:00 PM EDTs0 Wednesday, September 26, 2018 at 3:00:00 PM EDT R4 is our smallest server [storage used] and we had enough time to get it copied over while we were waiting on the SSDs to be transferred. The only reason we're doing these SSD transfers is because we're restoring servers as they finish copying to the SSDs. I'll detail all of this more once we're online 100% and I have a chance. 1 Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 About a half hour left of copying s2 from Slow Backup. R4 and R1 are restoring now. 2 Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 R2 is being copied from Slow Backup to Fast Backup in preparation for restoration. R4 Server finished restoring. R1 is almost done. S2 is restoring now. 2 Link to comment Share on other sites More sharing options...
ericr Posted September 25, 2018 Report Share Posted September 25, 2018 We are currently restoring S2 and R2. s1 Completed p1 Completed r1 Completed p2 Completed s2 In Progress r2 In Progresss3 Wednesday, September 26, 2018 at 12:00:00 AM EDTr3 Wednesday, September 26, 2018 at 8:00:00 AM EDTs4 Wednesday, September 26, 2018 at 12:00:00 PM EDTr4 Completed s5 Wednesday, September 26, 2018 at 5:00:00 PM EDTs0 Wednesday, September 26, 2018 at 5:00:00 PM EDT 1 Link to comment Share on other sites More sharing options...
ericr Posted September 25, 2018 Report Share Posted September 25, 2018 S2's restore is completed. Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 R2 is still restoring. S3 has about 2 hours before we are going to be able to start restoration by my estimation. Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 R2 server is almost done restoring and we're about to start on S3. We're running many restores in parallel and it's going quite quickly once we have the backups on the fast backup server. http://www.screen-shot.net/2018-09-25_10-46-49.png Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 We are now moving the SSDs from Slow Backup to Fast Backup to get started on restoring the S3 server. This is step 2 on the process outlined here: https://forums.mddhosting.com/topic/1583-major-outage-092118-client-discussion/?p=7288 1 Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 We are 75% restored overall. Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 R2 server is fully restored. Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 s1 Completedp1 Completedr1 Completedp2 Completeds2 Completedr2 Completeds3 Tuesday, September 25, 2018 at 7:00:00 PMr3 Wednesday, September 26, 2018 at 3:00:00 AMs4 Wednesday, September 26, 2018 at 4:00:00 AMr4 Completeds5 Wednesday, September 26, 2018 at 10:00:00 AMs0 Wednesday, September 26, 2018 at 10:00:00 AM 1 Link to comment Share on other sites More sharing options...
Michael D. Posted September 25, 2018 Author Report Share Posted September 25, 2018 S3 is in the process of being restored now - accounts are actively coming back online. Link to comment Share on other sites More sharing options...
ericr Posted September 25, 2018 Report Share Posted September 25, 2018 We are 72% of the way copying R3 off of the backup servver to the transfer ssd's There are 249 accounts left to restore on S3 1 Link to comment Share on other sites More sharing options...
Recommended Posts