Jump to content


Photo

Infrastructure Upgrades - Storage and Servers


  • Please log in to reply
55 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,665 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 August 2017 - 11:18 AM

Hello!
 

We have been working hard for about the last year to plan and execute some pretty huge upgrades for all of our customers.  For a couple of years now we've been using a Nimble CS500 SSD Accelerated Storage Area Network to provide highly available storage to our servers.  This worked great initially but as we've grown we've run into some issues here or there with the storage platform not meeting our expectations.

 

We are moving all servers to a StorPool Highly Available, Distributed, Self-Healing SSD Storage Area Network.  Our testing has shown that not only do we get more than 900% increase in I/O operations per second but we are also increasing the total bandwidth available to our storage platform by 1,200%.  While our Nimble Storage platform was SSD Cached where data read from the platform may or may not come from SSDs, all data read from our StorPool cluster will come from SSDs.

 

The end result of these migrations will be a faster and more consistent hosting experience across all of our hosting platforms - Shared, Reseller, Premium.  All VPS are already migrated to the new storage platform and all new orders for Shared, Reseller, and Premium hosting after August 15, 2017 are already on the new platform as well.

 

We will be migrating all servers over to the new storage platform on or before September 15, 2017 and we have already begun copying data in the background from the old platform to the new.  As a part of this migration we will be scheduling some downtime and we will post about it here as well as emailing all clients directly with details.

 

As we will be scheduling and announcing the downtime within maintenance windows and making you aware of these windows - we have a limited ability to migrate accounts manually prior to the maintenance window.  Please understand that while we understand nobody wants to have downtime - that we cannot possibly migrate everybody individually prior to the primary migrations and we will do our best in this regard.

 

If you have any questions or concerns about any of this please feel free to either reply to this thread or to open a support ticket directly.


  • 1

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!


#2 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,665 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 August 2017 - 11:51 AM

An email has been sent to all clients with an active shared, reseller, or premium hosting account about this upgrade and the migrations.


  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!


#3 AMC4x4

AMC4x4

    Newbie

  • Members
  • Pip
  • 14 posts

Posted 23 August 2017 - 11:56 AM

Every other hosting provider I've been with in the past would have allowed their clients to just have performance issues, and when they started complaining, they'd decide to do something about it then - reluctantly. Kudos to you for running your business differently, and for just wanting to make sure everything runs smoothly all the time. I have not noticed any impact to my sites. They've all been performing as expected, even when doing backups. Your post above, explaining all the details of this move - transparently - is why I have been with MDD for so long. It's why I trust my company's corporate site with you, and it's why I recommend you to everyone. Thank you, and keep up the great work. 


  • 0

#4 ComputerBob

ComputerBob

    Newbie

  • Members
  • Pip
  • 1 posts
  • Gender:Male

Posted 23 August 2017 - 12:14 PM

Thank you for that's excellent news, Mike -- just one more example of your proactive approach, explained so well by AMC4x4 above. :D


  • 0

#5 fjpoblam

fjpoblam

    Member

  • Members
  • PipPip
  • 54 posts

Posted 23 August 2017 - 12:43 PM

I've responded to the email, as I *think* was requested. A new ticket was created. I'm ready to help with my site whenever you folks are...
  • 0

#6 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,665 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 August 2017 - 12:52 PM

Thank you for your understanding on this - we wish we could do this without any sort of disruption or work on your part but such a large change necessitates it.

 

We're doing our best to minimize any impact from these moves and, as always, to be as transparent as we can be.  We can't promise there won't be any issues - but we're here to help if there are.


  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!


#7 SarisIsop

SarisIsop

    Member

  • Members
  • PipPip
  • 118 posts
  • Gender:Not Telling

Posted 23 August 2017 - 01:39 PM

Thank you for keeping us in the loop. I'm glad I have recently signed up for my 7th year with MDD.

 

:)

 

 


  • 0

#8 cziv

cziv

    Newbie

  • Members
  • Pip
  • 21 posts
  • Gender:Male

Posted 23 August 2017 - 03:10 PM

Hi.

 

Not sure what the email asked. To wait or to reply and have the plan moved earlier ?

 

For me it is the same. Still a puzzle what the email really means.

 

Thanks in any case for the quality of service.


  • 0

#9 Jeroen

Jeroen

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 23 August 2017 - 08:16 PM

Thank you for the notification.

 

Is there an indication of the downtime? We are probably talking about minutes, right?


  • 0

#10 rxpaul

rxpaul

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 24 August 2017 - 06:10 AM

Great to see you guys demonstrate the commitment & drive sorely missing from others. 

 

What would be the approximate downtime like (mins/hours/days)?


  • 0

#11 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,665 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 24 August 2017 - 02:03 PM

Hi.
 
Not sure what the email asked. To wait or to reply and have the plan moved earlier ?
 
For me it is the same. Still a puzzle what the email really means.
 
Thanks in any case for the quality of service.

You can wait, have some downtime during an off-peak time for your server in the future which we will email you about ahead of time, and have to do nothing [or] you can be migrated early and possibly have to update your mail clients / third party DNS [like CloudFlare], etc.

It's entirely up to you.
 

Thank you for the notification.
 
Is there an indication of the downtime? We are probably talking about minutes, right?


We're copying data with the servers online but as the server is online the data can change while we do this. Most notably would be MySQL - or users uploading files after we've already copied an account or folder, etc. So we are shutting down the old servers right after the online sync, doing an offline sync to make sure all of the data is intact and brought over, and then bringing the new copy online.

How long this takes really depends on how much data is on the server and what kind of transfer speeds we can get from our old platform - those are the limiting factors. We're in the process now of testing to get an idea of how long it'll take but it may be a couple of hours.
 
 

Great to see you guys demonstrate the commitment & drive sorely missing from others. 
 
What would be the approximate downtime like (mins/hours/days)?

It'll be as short as we can possibly make it - but due to the nature of how much data we're moving we are prioritizing data integrity over uptime in this case. We'll be updating this thread as well emailing all users additional details as we have them. We're still testing to get a good idea of how long the downtime will be.
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!


#12 perih

perih

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 24 August 2017 - 03:50 PM

This sounds like a fantastic upgrade, though I have no complaints about the existing service. Question: will this eliminate the need for my server data backups, particularly for email? Or will it still be necessary to find a way to ship data to another place on a daily basis? Thanks.


  • 0

#13 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 161 posts
  • Gender:Male

Posted 24 August 2017 - 05:02 PM

We always advise having backups that are not on our servers.  No matter how much we aim to provide layers of protection it is still possible for all those layers to fall apart and for you to lose data.

A specific example would be email.  As we only take backups nightly we would not be able to restore every email you received.  However, having your computers email client configured to do backups of all email received would provide that.


  • 0

#14 cziv

cziv

    Newbie

  • Members
  • Pip
  • 21 posts
  • Gender:Male

Posted 25 August 2017 - 06:41 AM

This sounds like a fantastic upgrade, though I have no complaints about the existing service. Question: will this eliminate the need for my server data backups, particularly for email? Or will it still be necessary to find a way to ship data to another place on a daily basis? Thanks.

 

Manually backup to an external place daily seems much of work. Personally i do that manually once per week. And once you create a backup in your plan root, there are plenty of software to somehow automate the download (ftp).


  • 0

#15 Jeroen

Jeroen

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 25 August 2017 - 06:16 PM

During the downtime, is it possible to have a maintenance page? And if yes, how should I do this (IP wise)


  • 0

#16 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 161 posts
  • Gender:Male

Posted 25 August 2017 - 07:35 PM

A maintenance page is not something that you would be able to setup as the server hosting content would be offline.  


  • 0

#17 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,665 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 25 August 2017 - 09:44 PM

We're going to have the servers offline for as little time as possible but how long it takes depends on how long it takes to sync any changed data. We hope for not more than 30 to 45 minutes but we are going to plan larger windows of time just in case.
  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!


#18 Jeroen

Jeroen

    Newbie

  • Members
  • Pip
  • 6 posts

Posted 26 August 2017 - 02:39 AM

We're going to have the servers offline for as little time as possible but how long it takes depends on how long it takes to sync any changed data. We hope for not more than 30 to 45 minutes but we are going to plan larger windows of time just in case.

Sounds good! I'm sure you'll do your best to make it as short as possible. Would be nice if there is a 503 status during the downtime :)


  • 0

#19 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 409 posts
  • Gender:Male

Posted 26 August 2017 - 11:55 AM

Sounds good! I'm sure you'll do your best to make it as short as possible. Would be nice if there is a 503 status during the downtime :)


I agree, but unfortunately the server will be completely offline and it wont be possible to show an error page.

You could redirect the site to a different server to display the error. Since we don't anticipate a long period of being offline, the site would probably be "down" longer if such an error page would be shown since it would need DNS propagation twice and the timing of starting the two DNS changes wouldn't be perfectly lined up wit the server transfers.
  • 0
Scott S - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Like us on Facebook: http://facebook.com/MDDHosting
http://www.mddhosting.com/ - Follow us on Twitter! @MDDHosting


#20 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,665 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 28 August 2017 - 03:09 PM

R3 Server Migration scheduled for 12 AM ET [GMT-4] on Saturday, September 2nd, 2017.

 

We are scheduling a 3 hour window from 12 AM to 3 AM ET although we expect the actual downtime to be approximately 45 minutes.

 

At approximately 12 AM we will be shutting down the server to do a final data sync from the old storage platform to the new storage platform to ensure there is no data loss or corruption.  Once this final offline data sync is completed the R3 server will be brought back online on the new storage.

 

During this outage all services on the server will be unavailable including cPanel, WHM, FTP, IMAP, POP3, SMTP, and HTTP/HTTPS.  We will have the services restored as quickly as humanly possible.

 

We are not wiping or erasing the old storage platform so in a worst-case scenario the old copy of R3 will be brought back online on the old storage platform.

 

All clients affected by this scheduled maintenance have been notified via email.

 

If you have any questions or concerns about this do please let us know.


  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users