Michael D. Posted August 23, 2017 Report Share Posted August 23, 2017 Hello! We have been working hard for about the last year to plan and execute some pretty huge upgrades for all of our customers. For a couple of years now we've been using a Nimble CS500 SSD Accelerated Storage Area Network to provide highly available storage to our servers. This worked great initially but as we've grown we've run into some issues here or there with the storage platform not meeting our expectations. We are moving all servers to a StorPool Highly Available, Distributed, Self-Healing SSD Storage Area Network. Our testing has shown that not only do we get more than 900% increase in I/O operations per second but we are also increasing the total bandwidth available to our storage platform by 1,200%. While our Nimble Storage platform was SSD Cached where data read from the platform may or may not come from SSDs, all data read from our StorPool cluster will come from SSDs. The end result of these migrations will be a faster and more consistent hosting experience across all of our hosting platforms - Shared, Reseller, Premium. All VPS are already migrated to the new storage platform and all new orders for Shared, Reseller, and Premium hosting after August 15, 2017 are already on the new platform as well. We will be migrating all servers over to the new storage platform on or before September 15, 2017 and we have already begun copying data in the background from the old platform to the new. As a part of this migration we will be scheduling some downtime and we will post about it here as well as emailing all clients directly with details. As we will be scheduling and announcing the downtime within maintenance windows and making you aware of these windows - we have a limited ability to migrate accounts manually prior to the maintenance window. Please understand that while we understand nobody wants to have downtime - that we cannot possibly migrate everybody individually prior to the primary migrations and we will do our best in this regard. If you have any questions or concerns about any of this please feel free to either reply to this thread or to open a support ticket directly. 1 Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 23, 2017 Author Report Share Posted August 23, 2017 An email has been sent to all clients with an active shared, reseller, or premium hosting account about this upgrade and the migrations. Quote Link to comment Share on other sites More sharing options...
AMC4x4 Posted August 23, 2017 Report Share Posted August 23, 2017 Every other hosting provider I've been with in the past would have allowed their clients to just have performance issues, and when they started complaining, they'd decide to do something about it then - reluctantly. Kudos to you for running your business differently, and for just wanting to make sure everything runs smoothly all the time. I have not noticed any impact to my sites. They've all been performing as expected, even when doing backups. Your post above, explaining all the details of this move - transparently - is why I have been with MDD for so long. It's why I trust my company's corporate site with you, and it's why I recommend you to everyone. Thank you, and keep up the great work. Quote Link to comment Share on other sites More sharing options...
ComputerBob Posted August 23, 2017 Report Share Posted August 23, 2017 Thank you for that's excellent news, Mike -- just one more example of your proactive approach, explained so well by AMC4x4 above. Quote Link to comment Share on other sites More sharing options...
fjpoblam Posted August 23, 2017 Report Share Posted August 23, 2017 I've responded to the email, as I *think* was requested. A new ticket was created. I'm ready to help with my site whenever you folks are... Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 23, 2017 Author Report Share Posted August 23, 2017 Thank you for your understanding on this - we wish we could do this without any sort of disruption or work on your part but such a large change necessitates it. We're doing our best to minimize any impact from these moves and, as always, to be as transparent as we can be. We can't promise there won't be any issues - but we're here to help if there are. Quote Link to comment Share on other sites More sharing options...
SarisIsop Posted August 23, 2017 Report Share Posted August 23, 2017 Thank you for keeping us in the loop. I'm glad I have recently signed up for my 7th year with MDD. Quote Link to comment Share on other sites More sharing options...
cziv Posted August 23, 2017 Report Share Posted August 23, 2017 Hi. Not sure what the email asked. To wait or to reply and have the plan moved earlier ? For me it is the same. Still a puzzle what the email really means. Thanks in any case for the quality of service. Quote Link to comment Share on other sites More sharing options...
Jeroen Posted August 24, 2017 Report Share Posted August 24, 2017 Thank you for the notification. Is there an indication of the downtime? We are probably talking about minutes, right? Quote Link to comment Share on other sites More sharing options...
rxpaul Posted August 24, 2017 Report Share Posted August 24, 2017 Great to see you guys demonstrate the commitment & drive sorely missing from others. What would be the approximate downtime like (mins/hours/days)? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 24, 2017 Author Report Share Posted August 24, 2017 Hi. Not sure what the email asked. To wait or to reply and have the plan moved earlier ? For me it is the same. Still a puzzle what the email really means. Thanks in any case for the quality of service.You can wait, have some downtime during an off-peak time for your server in the future which we will email you about ahead of time, and have to do nothing [or] you can be migrated early and possibly have to update your mail clients / third party DNS [like CloudFlare], etc. It's entirely up to you. Thank you for the notification. Is there an indication of the downtime? We are probably talking about minutes, right? We're copying data with the servers online but as the server is online the data can change while we do this. Most notably would be MySQL - or users uploading files after we've already copied an account or folder, etc. So we are shutting down the old servers right after the online sync, doing an offline sync to make sure all of the data is intact and brought over, and then bringing the new copy online. How long this takes really depends on how much data is on the server and what kind of transfer speeds we can get from our old platform - those are the limiting factors. We're in the process now of testing to get an idea of how long it'll take but it may be a couple of hours. Great to see you guys demonstrate the commitment & drive sorely missing from others. What would be the approximate downtime like (mins/hours/days)?It'll be as short as we can possibly make it - but due to the nature of how much data we're moving we are prioritizing data integrity over uptime in this case. We'll be updating this thread as well emailing all users additional details as we have them. We're still testing to get a good idea of how long the downtime will be. Quote Link to comment Share on other sites More sharing options...
perih Posted August 24, 2017 Report Share Posted August 24, 2017 This sounds like a fantastic upgrade, though I have no complaints about the existing service. Question: will this eliminate the need for my server data backups, particularly for email? Or will it still be necessary to find a way to ship data to another place on a daily basis? Thanks. Quote Link to comment Share on other sites More sharing options...
ericr Posted August 24, 2017 Report Share Posted August 24, 2017 We always advise having backups that are not on our servers. No matter how much we aim to provide layers of protection it is still possible for all those layers to fall apart and for you to lose data.A specific example would be email. As we only take backups nightly we would not be able to restore every email you received. However, having your computers email client configured to do backups of all email received would provide that. Quote Link to comment Share on other sites More sharing options...
cziv Posted August 25, 2017 Report Share Posted August 25, 2017 This sounds like a fantastic upgrade, though I have no complaints about the existing service. Question: will this eliminate the need for my server data backups, particularly for email? Or will it still be necessary to find a way to ship data to another place on a daily basis? Thanks. Manually backup to an external place daily seems much of work. Personally i do that manually once per week. And once you create a backup in your plan root, there are plenty of software to somehow automate the download (ftp). Quote Link to comment Share on other sites More sharing options...
Jeroen Posted August 25, 2017 Report Share Posted August 25, 2017 During the downtime, is it possible to have a maintenance page? And if yes, how should I do this (IP wise) Quote Link to comment Share on other sites More sharing options...
ericr Posted August 26, 2017 Report Share Posted August 26, 2017 A maintenance page is not something that you would be able to setup as the server hosting content would be offline. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 26, 2017 Author Report Share Posted August 26, 2017 We're going to have the servers offline for as little time as possible but how long it takes depends on how long it takes to sync any changed data. We hope for not more than 30 to 45 minutes but we are going to plan larger windows of time just in case. Quote Link to comment Share on other sites More sharing options...
Jeroen Posted August 26, 2017 Report Share Posted August 26, 2017 We're going to have the servers offline for as little time as possible but how long it takes depends on how long it takes to sync any changed data. We hope for not more than 30 to 45 minutes but we are going to plan larger windows of time just in case.Sounds good! I'm sure you'll do your best to make it as short as possible. Would be nice if there is a 503 status during the downtime Quote Link to comment Share on other sites More sharing options...
Scott Posted August 26, 2017 Report Share Posted August 26, 2017 Sounds good! I'm sure you'll do your best to make it as short as possible. Would be nice if there is a 503 status during the downtime I agree, but unfortunately the server will be completely offline and it wont be possible to show an error page. You could redirect the site to a different server to display the error. Since we don't anticipate a long period of being offline, the site would probably be "down" longer if such an error page would be shown since it would need DNS propagation twice and the timing of starting the two DNS changes wouldn't be perfectly lined up wit the server transfers. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 28, 2017 Author Report Share Posted August 28, 2017 R3 Server Migration scheduled for 12 AM ET [GMT-4] on Saturday, September 2nd, 2017. We are scheduling a 3 hour window from 12 AM to 3 AM ET although we expect the actual downtime to be approximately 45 minutes. At approximately 12 AM we will be shutting down the server to do a final data sync from the old storage platform to the new storage platform to ensure there is no data loss or corruption. Once this final offline data sync is completed the R3 server will be brought back online on the new storage. During this outage all services on the server will be unavailable including cPanel, WHM, FTP, IMAP, POP3, SMTP, and HTTP/HTTPS. We will have the services restored as quickly as humanly possible. We are not wiping or erasing the old storage platform so in a worst-case scenario the old copy of R3 will be brought back online on the old storage platform. All clients affected by this scheduled maintenance have been notified via email. If you have any questions or concerns about this do please let us know. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 2, 2017 Author Report Share Posted September 2, 2017 Just a reminder - the maintenance window for the R3 server starts in just a little over 2.5 hours. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 2, 2017 Author Report Share Posted September 2, 2017 The online syncs are completing faster than expected. We're hoping that the offline one goes even faster. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 2, 2017 Author Report Share Posted September 2, 2017 We are about to get started now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 2, 2017 Author Report Share Posted September 2, 2017 The R3 server has been taken offline for the offline sync of data. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted September 2, 2017 Author Report Share Posted September 2, 2017 The offline copy is still running at this point. We have run into a couple of issues we didn't expect but we've adjusted for them. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.