Jump to content


Photo

Major Outage - 09/21/18 - 09/24/2018


  • This topic is locked This topic is locked
97 replies to this topic

#21 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 10:36 AM

I'm still watching the transfer progress and we are doing our best to keep up with support tickets.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#22 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 10:42 AM

The copy of the backup for the first server, S1, is done - we are now going to be restoring that data onto the much improved backup system and beginning restores right to the server.  As this is from SSD to SSD to SSD it should all be very fast but I will keep you informed.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#23 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 01:37 PM

We are working on getting the first server backups ready to restore directly to the server and are preparing to start copying the second server.  The second server is going to be P1, the third server is going to be R1.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#24 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 01:39 PM

I and many of my clients are on s1.

 

The cPanel error page has gone, but I still can't get to the sites. I now get a different error page saying my browser can't establish a secure connection. I tried http: instead of https: but it made no difference.

 

When I try to open cPanel, I'm told my login is invalid.

 

I'm hoping these are just symptoms of the recovery process, and that once the sites are fully recovered I'll have access again.

 

Of course I'm impatient like everyone else, but like many of the members above, I've been nothing but delighted with MDD over the years and have no intention of changing hosts.

 

Mark

We changed the text of the error page at the request of quite a few clients so that their visitors would know that they weren't hacked and will come back as soon as possible.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#25 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 02:54 PM

The backup of S1 is copying to our 24 Disk SSD Array from the 4-Disk array and once this is done we will begin restorations of the S1 server - it should go very quickly.

 

We are presently copying from the old backup server to another set of 4 SSDs the backup for P1.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#26 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 03:22 PM

The S1 backup image is still copying to the 24 Drive SSD array for restoration.  The P1 backup image is being created on the 4 Disk SSD array in preparation for copying to the 24 Drive system.


  • 1
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#27 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 06:54 PM

We are now restoring backups for the S1 server - this new backup server is going EXCEPTIONALLY fast so far in comparison to restorations from the old backup server.

 

Once we have gotten more accounts restored we'll give our best ETA we can come up with on S1.

 

We are still copying the P1 backup at this time and will be starting on the R1 backup shortly.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#28 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 06:56 PM

S1 restorations are in the accounts starting with BN.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#29 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 06:58 PM

The restorations from the new backup system to the S1 server are going about 1000% faster than the old backup server by my estimations.  I'm still trying to work out a good ETA - and I need to wait on some more accounts to restore before I have a data set big enough to make such an estimation.


  • 2
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#30 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 07:09 PM

We are restoring accounts at about the rate of 900 per hour from the new backup server and have approximately 9,000 accounts to restore.

 

Assuming everything scales smoothly we should be able to have everything restored within 10 to 20 hours from this point.  This is a very tentative ETA.

 

It is possible it may still take up to 48 hours for 100% as it will still take time to copy the full backups off of the old backup server and onto the new one and due to the nature of the old backup server we can only do one at a time.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#31 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 07:31 PM

Here are the current calculated ETA's:

 

p1    Monday, September 24, 2018 at 1:00:00 AM
r1    Monday, September 24, 2018 at 10:00:00 AM
s2    Monday, September 24, 2018 at 8:00:00 PM
p2    Tuesday, September 25, 2018 at 1:00:00 AM
r2    Tuesday, September 25, 2018 at 10:00:00 AM
s3    Tuesday, September 25, 2018 at 10:00:00 PM
r3    Wednesday, September 26, 2018 at 6:00:00 AM
s4    Wednesday, September 26, 2018 at 4:00:00 PM
r4    Wednesday, September 26, 2018 at 10:00:00 PM
s5    Thursday, September 27, 2018 at 7:00:00 AM
s0    Thursday, September 27, 2018 at 1:00:00 PM

 

These are likely over-estimations [i.e. should be restored before this] and we feel that this is likely worst case.  The reason we won't be done in as long as it would simply take to restore the data to the server is because we're strapped down by the transfer rate between the old backup server and the new one with it taking about 8 hours to transfer ~3TB of data.

 

These times are for 100% restoration of all services.

 

We will update this thread if this changes.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#32 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 08:58 PM

Latest ETA Calculations:

 

s1    Sunday, September 23, 2018 at 11:59:00 PM
p1    Monday, September 24, 2018 at 4:00:00 AM
r1    Monday, September 24, 2018 at 4:00:00 PM
s2    Monday, September 24, 2018 at 4:00:00 PM
p2    Monday, September 24, 2018 at 3:00:00 PM
r2    Tuesday, September 25, 2018 at 2:00:00 AM
s3    Tuesday, September 25, 2018 at 6:00:00 AM
r3    Tuesday, September 25, 2018 at 2:00:00 PM
s4    Tuesday, September 25, 2018 at 5:00:00 PM
r4    Tuesday, September 25, 2018 at 6:00:00 PM
s5    Tuesday, September 25, 2018 at 10:00:00 PM
s0    Tuesday, September 25, 2018 at 10:00:00 PM


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#33 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 219 posts
  • Gender:Male

Posted 23 September 2018 - 10:55 PM

We had to remove the ETA as there was a mistake made on the sheet causing the ETA to be off by a few years(we do not even remotely plan on it taking till march 2019).  I do apologize. As for S2 it is still 4th. 

 
s1 Monday, September 24, 2018 at 2:00:00 AM
p1 Monday, September 24, 2018 at 4:00:00 AM
r1 Monday, September 24, 2018 at 6:00:00 PM
s2 Monday, September 24, 2018 at 7:00:00 PM
p2 Monday, September 24, 2018 at 6:00:00 PM
r2 Tuesday, September 25, 2018 at 5:00:00 AM
s3 Tuesday, September 25, 2018 at 12:00:00 PM
r3 Tuesday, September 25, 2018 at 8:00:00 PM
s4 Wednesday, September 26, 2018 at 1:00:00 AM
r4 Wednesday, September 26, 2018 at 2:00:00 AM
s5 Wednesday, September 26, 2018 at 8:00:00 AM
s0 Wednesday, September 26, 2018 at 8:00:00 AM

  • 0

#34 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 September 2018 - 11:02 PM

Here are the latest ETA's. The transfers from the old backup server to the new one are taking a little longer than expected. All times are Eastern Daylight Time.
 

s1 Monday, September 24, 2018 at 2:00:00 AM
p1 Monday, September 24, 2018 at 4:00:00 AM
r1 Monday, September 24, 2018 at 6:00:00 PM
s2 Monday, September 24, 2018 at 7:00:00 PM
p2 Monday, September 24, 2018 at 6:00:00 PM
r2 Tuesday, September 25, 2018 at 5:00:00 AM
s3 Tuesday, September 25, 2018 at 12:00:00 PM
r3 Tuesday, September 25, 2018 at 8:00:00 PM
s4 Wednesday, September 26, 2018 at 1:00:00 AM
r4 Wednesday, September 26, 2018 at 2:00:00 AM
s5 Wednesday, September 26, 2018 at 8:00:00 AM
s0 Wednesday, September 26, 2018 at 8:00:00 AM


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#35 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 12:14 AM

The R1 server is now being copied from Slow Backup to Fast Backup.

The Restoration of accounts from the P1 backups on Fast Backup are starting now.

 

This is fast backup:

 

Attached File  2018-09-24_01-11-07.jpg   93.41KB   0 downloads

 

It is connected via 10GBPS Networking and is running 24 x 1.6TB SSDs via Hardware Raid.  The four loose drives in the top are 4 out of 8 SSDs we are using to transfer data from Slow Backup to Fast Backup.  This is a temporary solution to get everybody restored as quickly as possible.  Our provider in Phoenix, IOFlood, is helping us out with this recovery by letting us use this hardware to speed up the process.  They're also handling all of the disk swaps to transfer data from the Slow Backup to the Fast Backup.  We couldn't be doing this without them.

 

Here is fast backup hard at work restoring client data:

 

Here's Slow Backup

Attached File  2018-09-24_01-20-20.jpg   39.43KB   0 downloads


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#36 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,889 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 September 2018 - 12:33 AM

I am going to get a little bit of sleep - Eric will keep updates posted here on the forums and we're still working on support tickets as quickly as we can.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#37 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 219 posts
  • Gender:Male

Posted 24 September 2018 - 02:23 AM

Status update:
All accounts on S1 are restored. 

The p1 archive is 24% of the way of being restored onto the 10G ssd server.

The r1 backup is about halfway done being extracted from the backup server in preparation to being restored on the 10G ssd server.


  • 0

#38 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 219 posts
  • Gender:Male

Posted 24 September 2018 - 04:08 AM

We were doing restores differently initially and that process was significantly slower.  We are not overwriting any accounts unless specifically told to. 


  • 0

#39 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 219 posts
  • Gender:Male

Posted 24 September 2018 - 05:23 AM

I have started the restoration of p1's accounts at this time.


  • 0

#40 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 219 posts
  • Gender:Male

Posted 24 September 2018 - 08:38 AM

With the exception of one single account P1 is done with it's restorations.  The account that we are waiting on is on the larger side so I did not want to save the status update on it.


  • 0




1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users