Jump to content


Photo

Infrastructure Upgrades - Storage and Servers


  • Please log in to reply
55 replies to this topic

#21 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 01 September 2017 - 08:21 PM

Just a reminder - the maintenance window for the R3 server starts in just a little over 2.5 hours.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#22 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 01 September 2017 - 10:37 PM

The online syncs are completing faster than expected.  We're hoping that the offline one goes even faster.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#23 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 01 September 2017 - 10:59 PM

We are about to get started now.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#24 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 01 September 2017 - 11:05 PM

The R3 server has been taken offline for the offline sync of data.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#25 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 01 September 2017 - 11:28 PM

The offline copy is still running at this point.  We have run into a couple of issues we didn't expect but we've adjusted for them.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#26 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 02 September 2017 - 12:01 AM

CageFS, a technology we use to isolate user accounts from each other, is causing the offline transfer to take longer than anticipated.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#27 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 02 September 2017 - 01:05 AM

The offline sync is taking longer than expected due to virtual file systems and some files and folders we neglected to copy during the online sync.  We'll continue to keep this thread updated.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#28 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 223 posts
  • Gender:Male

Posted 02 September 2017 - 02:12 AM

To be clear, the home directory was not cleanly copied over in the sync we did before we took the server offline.  Right now I am correcting the problems and will update as soon as I have more news.


  • 0

#29 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 223 posts
  • Gender:Male

Posted 02 September 2017 - 02:18 AM

The rsync is not currently finding missing files.  This is good news.  I will update further as soon as possible.


  • 0

#30 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 223 posts
  • Gender:Male

Posted 02 September 2017 - 03:19 AM

I have found more directories with issues.  I will update when these copies are complete.


  • 0

#31 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 223 posts
  • Gender:Male

Posted 02 September 2017 - 03:59 AM

We are about 2/3rds through the home directory.


  • 0

#32 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 223 posts
  • Gender:Male

Posted 02 September 2017 - 04:14 AM

I have aborted the work.  We will be rescheduling the work for next weekend after we reasses the procedure.


  • 0

#33 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 02 September 2017 - 09:27 AM

We will be re-evaluating migrating the server as a whole.  The plan was for it to take no more than a couple of hours and for no IP addresses or Server Names to change.  Unfortunately this method may not work and we may be forced to perform standard cPanel migrations.

 

We were able to use this method to migrate all of our internal servers and systems but our old storage platform simply may not be able to keep up with the amount of data transfer required to migrate one of these servers as a whole.

 

If this is the ultimate outcome we will work with you as much as possible to make the transfer process as smooth as possible.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#34 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 07 September 2017 - 10:37 AM

I hate to do it - but we're scheduling another maintenance window for the R3 server this Friday, September 8th, 2017 from 10 PM ET to 2 AM ET [GMT-4].  The last failed attempted did not take very long to copy data we actually spent a fair bit of the time attempting to get the new server to boot successfully.  We have performed extensive testing since the transfer failure of the R3 server this last Saturday and we are confident that we have resolved all of the outstanding issues that prevented the transfer from succeeding.

 

We plan for the actual outage to be less than an hour, however, we will keep this thread updated as we did with the last attempt.

 

I did intend to send out an email on Tuesday, as Monday was a holiday, detailing the failure and what we're doing to correct it but admittedly I've been so focused on getting the issues resolved that it slipped my mind.  I am sending out an email now to all affected clients about this.

 

If you have any questions you can ask them here or reply to the email you're about to receive, if you're on the R3 server, and we'll be happy to help.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#35 Jeroen

Jeroen

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 08 September 2017 - 01:14 AM

Not my server, but still GOOD LUCK!


  • 0

#36 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 08 September 2017 - 08:42 PM

We will be starting maintenance on the R3 server in about 20 minutes.
  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#37 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 223 posts
  • Gender:Male

Posted 08 September 2017 - 09:00 PM

I have brought R3 offline and I am starting the offline synchronization process.


  • 0

#38 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 223 posts
  • Gender:Male

Posted 08 September 2017 - 10:07 PM

We are making good strides.  We are about halfway through the home dir after a 30 minute delay caused by the vmware cluster re-configuring the network interfaces unexpectedly.


  • 0

#39 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 08 September 2017 - 10:32 PM

The data sync is still in progress.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#40 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 223 posts
  • Gender:Male

Posted 08 September 2017 - 10:38 PM

The home directory is synchronized. The process has completed cagefs as well and is now working on mysql.


  • 0




5 user(s) are reading this topic

0 members, 5 guests, 0 anonymous users