Jump to content
MDDHosting Forums

Major Outage - 09/21/18 - 09/24/2018


Recommended Posts

Some accounts on S1 were restored without MySQL databases - I don't know why this is as of yet as every backup includes database backups - but if you find your databases aren't listed you can restore them via cPanel -> JetBackup -> Database Backups.

 

After doing so you will also need to go to cPanel -> MySQL Databases and re-create your database user and apply it to your databases.

 

We're happy to help you with it if you need it in a ticket and we're doing our best to keep up.

Link to comment
Share on other sites

The copy of data for the S2 server from Slow Backup to Intermediate Transfer DIsks is about 75% completed. Once it's 100% we'll move the intermediate disks over to FastBackup and get it ready for restoration.

 

1 hour 30 minutes left preparing R1 for restoration from Fast Backup.

  • Upvote 2
Link to comment
Share on other sites

We upgraded a storage component on the new Fast Backup server and we are seeing a huge increase in performance over what we were already seeing. This should cut down the total restoration time - we will have updated ETAs once we have conducted some more transfers.

  • Upvote 5
Link to comment
Share on other sites

We performed some changes on the storage in both backup servers. The fast backup server is now doing 4,770 GB/hour data rates while it was doing 500gb/hour. The old backup server has been doing 318 gb/hour. We're watching to see how much of an improvement the old backup server gets as a result of these changes too.

 

The bottle neck is getting the data from the old backup server to the fast one - but this process is substantially faster than restoring directly from the old backup server which was only doing about 50gb/hour.

  • Upvote 5
Link to comment
Share on other sites

We are over 50% restored network-wide. 51.4% actually. We are still hoping to be 100% by tomorrow - but the ETAs are calculations based upon transfer speeds and restoration rates. As you've seen they fluctuate and we're still honestly hoping to be one far sooner than any of the ETAs say.

  • Upvote 1
Link to comment
Share on other sites

Latest ETAs:

 

 

s1 Completed
p1 Completed
r1 In Progress
p2 Completed
s2 Tuesday, September 25, 2018 at 6:00:00 AM EDT
r2 Tuesday, September 25, 2018 at 2:00:00 PM EDT
s3 Tuesday, September 25, 2018 at 9:00:00 PM EDT
r3 Wednesday, September 26, 2018 at 5:00:00 AM EDT
s4 Wednesday, September 26, 2018 at 9:00:00 AM EDT
r4 In progress
s5 Wednesday, September 26, 2018 at 3:00:00 PM EDT
s0 Wednesday, September 26, 2018 at 3:00:00 PM EDT

 

R4 is our smallest server [storage used] and we had enough time to get it copied over while we were waiting on the SSDs to be transferred. The only reason we're doing these SSD transfers is because we're restoring servers as they finish copying to the SSDs.

 

I'll detail all of this more once we're online 100% and I have a chance.

  • Upvote 1
Link to comment
Share on other sites

We are currently restoring S2 and R2.

 

s1 Completed
p1 Completed
r1 Completed
p2 Completed
s2 In Progress
r2 In Progress
s3 Wednesday, September 26, 2018 at 12:00:00 AM EDT
r3 Wednesday, September 26, 2018 at 8:00:00 AM EDT
s4 Wednesday, September 26, 2018 at 12:00:00 PM EDT
r4 Completed
s5 Wednesday, September 26, 2018 at 5:00:00 PM EDT
s0 Wednesday, September 26, 2018 at 5:00:00 PM EDT

 

 

  • Upvote 1
Link to comment
Share on other sites

s1 Completed
p1 Completed
r1 Completed
p2 Completed
s2 Completed
r2 Completed
s3 Tuesday, September 25, 2018 at 7:00:00 PM
r3 Wednesday, September 26, 2018 at 3:00:00 AM
s4 Wednesday, September 26, 2018 at 4:00:00 AM
r4 Completed
s5 Wednesday, September 26, 2018 at 10:00:00 AM
s0 Wednesday, September 26, 2018 at 10:00:00 AM

  • Upvote 1
Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...