Jump to content


Photo

Major Outage - 09/21/18 - 09/24/2018


  • This topic is locked This topic is locked
97 replies to this topic

#61 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 07:52 PM

We are over 50% restored network-wide. 51.4% actually.  We are still hoping to be 100% by tomorrow - but the ETAs are calculations based upon transfer speeds and restoration rates.  As you've seen they fluctuate and we're still honestly hoping to be one far sooner than any of the ETAs say.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#62 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 07:54 PM

We are restoring R1 still as well as R4.  Accounts are actively coming online and will keep doing so until these servers are fully restored.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#63 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 08:04 PM

Latest ETAs:

s1    Completed    
p1    Completed    
r1    In Progress    
p2    Completed    
s2    Tuesday, September 25, 2018 at 6:00:00 AM    EDT
r2    Tuesday, September 25, 2018 at 2:00:00 PM    EDT
s3    Tuesday, September 25, 2018 at 9:00:00 PM    EDT
r3    Wednesday, September 26, 2018 at 5:00:00 AM    EDT
s4    Wednesday, September 26, 2018 at 9:00:00 AM    EDT
r4    In progress    
s5    Wednesday, September 26, 2018 at 3:00:00 PM    EDT
s0    Wednesday, September 26, 2018 at 3:00:00 PM    EDT

 

R4 is our smallest server [storage used] and we had enough time to get it copied over while we were waiting on the SSDs to be transferred.  The only reason we're doing these SSD transfers is because we're restoring servers as they finish copying to the SSDs.

 

I'll detail all of this more once we're online 100% and I have a chance.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#64 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 09:06 PM

About a half hour left of copying s2 from Slow Backup.  R4 and R1 are restoring now.


  • 2
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#65 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 10:26 PM

R2 is being copied from Slow Backup to Fast Backup in preparation for restoration.

 

R4 Server finished restoring.

 

R1 is almost done.

 

S2 is restoring now.


  • 2
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#66 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 25 September 2018 - 04:44 AM

We are currently restoring S2 and R2.

 

s1 Completed
p1 Completed
r1 Completed
p2 Completed
s2 In Progress
r2 In Progress
s3 Wednesday, September 26, 2018 at 12:00:00 AM EDT
r3 Wednesday, September 26, 2018 at 8:00:00 AM EDT
s4 Wednesday, September 26, 2018 at 12:00:00 PM EDT
r4 Completed
s5 Wednesday, September 26, 2018 at 5:00:00 PM EDT
s0 Wednesday, September 26, 2018 at 5:00:00 PM EDT

 

 


  • 1

#67 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 25 September 2018 - 06:36 AM

S2's restore is completed.  


  • 0

#68 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 08:15 AM

R2 is still restoring.  S3 has about 2 hours before we are going to be able to start restoration by my estimation.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#69 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 09:50 AM

R2 server is almost done restoring and we're about to start on S3.

 

We're running many restores in parallel and it's going quite quickly once we have the backups on the fast backup server.

 

2018-09-25_10-46-49.png


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#70 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 10:36 AM

We are now moving the SSDs from Slow Backup to Fast Backup to get started on restoring the S3 server.  This is step 2 on the process outlined here: https://forums.mddho...cussion/?p=7288


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#71 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 10:43 AM

We are 75% restored overall.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#72 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 10:52 AM

R2 server is fully restored.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#73 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 11:14 AM

s1    Completed
p1    Completed
r1    Completed
p2    Completed
s2    Completed
r2    Completed
s3    Tuesday, September 25, 2018 at 7:00:00 PM
r3    Wednesday, September 26, 2018 at 3:00:00 AM
s4    Wednesday, September 26, 2018 at 4:00:00 AM
r4    Completed
s5    Wednesday, September 26, 2018 at 10:00:00 AM
s0    Wednesday, September 26, 2018 at 10:00:00 AM


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#74 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 12:37 PM

S3 is in the process of being restored now - accounts are actively coming back online.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#75 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 25 September 2018 - 01:29 PM

We are 72% of the way copying R3 off of the backup servver to the transfer ssd's

 

There are 249 accounts left to restore on S3


  • 1

#76 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 25 September 2018 - 04:07 PM

We are now starting the restore of R3

 

s1    Completed    
p1    Completed    
r1    Completed    
p2    Completed    
s2    Completed    
r2    Completed    
s3    In Progress    
r3    In Progress    
s4    Wednesday, September 26, 2018 at 5:00:00 AM    EDT
r4    Completed    
s5    Wednesday, September 26, 2018 at 11:00:00 AM    EDT
s0    Wednesday, September 26, 2018 at 11:00:00 AM    EDT

  • 1

#77 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 04:15 PM

80% Restored/Back Online.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#78 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 25 September 2018 - 04:56 PM

Due to a unexpected reboot of the backup server we are going to have to swap s5 and s4 so we can not lose progress made towards s4's restoration.


  • 0

#79 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 07:14 PM

R3 is about 50% restored - it's taking longer than expected as it as a lot of tiny accounts.  S3 is just about done.  S5 will be starting soon.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#80 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 September 2018 - 07:27 PM

The S3 server is fully restored.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users