Michael D. Posted October 18, 2018 Report Share Posted October 18, 2018 Why we're upgrading processors. The new processors will have the same number of cores and threads as our old processors, however, will be clocked 21% faster at 2.3 GHz in comparison to the 1.8 GHz processors currently in our compute nodes. Originally when we set up our compute cluster we were being billed on power by our actual metered usage and, as such, we went with low-power-consumption processors to keep our power usage down. We are now being billed by by the circuit meaning that regardless of how much power we use - we pay the same amount so long as we aren't using more than the power circuit can provide. The new processors we will be rolling out are rated at 120 Watts versus the 65 Watts our current processors use - using around 45% more power per processor. While it's less important for a server than a desktop or laptop - the current processors have a turbo frequency of 2.5 GHz and the new processors turbo at 3.1 GHz. Servers do not get to use Turbo frequencies as often as desktops and laptops due to their fairly constant workload but it does happen occasionally. When the server hosting *your* site be upgraded. We will be placing our heaviest servers on the new processors first as they have the most client accounts and we want as many people to benefit from the upgrades as possible. We always watch our resource usage carefully - especially CPU - and will lean towards leaving breathing room for the servers to ensure smooth and stable operation post-upgrade. As of right now most likely the S1, P1, and R1 servers will be placed on the upgraded processors when the first upgrades are completed. We are going to evaluate the CPU usage once these servers are on the new processors to evaluate whether or not we can move any additional hosting servers to the new processors during the first upgrade. We'd will lean towards not overloading the servers and ensuring stable performance. We will be upgrading all compute nodes as soon as we're able to do so and, as a result, everybody will be getting upgraded at some point in the near future. We have been planning on upgrading the processors in our compute nodes for some time now. We're a small provider and we do our best to continue investing in our platform to increase performance and reliability. Unfortunately upgrading the processors is not a cheap endeavor and is something that we're going to have to work on over time. How the maintenance will be performed. The plan is to perform the upgrades themselves one hardware node at a time. We may need to shut down and boot back up some hosting servers as we move them around during the upgrades and this process takes approximately 30 to 60 seconds per hosting server. While we can live-migrate the hosting servers we have found that due to how busy our hosting servers are the live migration can actually impact performance much longer than the 30 to 60 seconds a shut down and boot-up can take. In short we've found it's better to shut down and boot up than to live migrate from an actual ability-to-use-the-service standpoint. Due to the extremely short duration of any downtime that will occur with the servers the lead-time on the restarts may be shorter than usual. We will update this thread with details as we perform the upgrades and will also email anybody on a server that is being upgraded prior to performing the work. If you have any questions about any of this feel free to reply to this thread or to open a support ticket referencing this thread. Thank you! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 20, 2018 Author Report Share Posted October 20, 2018 We're performing tests on the first node to be upgraded to ensure there are no issues. Hardware burn-in testing is always a good idea - if there are any issues we want to find them before they have a chance to affect our clients. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 20, 2018 Author Report Share Posted October 20, 2018 We will be shutting down the S1 server this evening for approximately 60 seconds to move it onto a hardware node with the upgraded processors. We will update this thread just prior to doing this and immediately after. The current time scheduled to do this is approximately 10 PM ET [a little over 8 hours from now]. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 21, 2018 Author Report Share Posted October 21, 2018 We are shutting the S1 server down now to move it to the new processors . Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 21, 2018 Author Report Share Posted October 21, 2018 The server is coming back online now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 21, 2018 Author Report Share Posted October 21, 2018 All is well with S1 - it was successfully moved to the new processors/hardware . Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 21, 2018 Author Report Share Posted October 21, 2018 From the looks of it - this upgrade is going to carry an even bigger performance gain than anticipated. We'll obviously need some more time to collect data - but it's pretty obvious the before and after on this graph. http://www.screen-shot.net/2018-10-20_22-47-31.png http://www.screen-shot.net/2018-10-20_22-47-47.png PHP Execution looks to be happening approximately twice as fast. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 21, 2018 Author Report Share Posted October 21, 2018 Yeah, I am definitely surprised at how large of an improvement this is. We will be upgrading all servers as quickly as we can. The original goal was to upgrade all servers at once but due to the major disaster and the damage it caused to our revenue we're having to do this in stages. http://www.screen-shot.net/2018-10-20_22-56-17.png 1 Quote Link to comment Share on other sites More sharing options...
008Rohit Posted October 21, 2018 Report Share Posted October 21, 2018 Hi, Will this have any impact on the loading times of cached pages (i.e. pages that don't require PHP execution)? Also, any plans on upgrading your storage to NVMe SSDs in the near future? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 21, 2018 Author Report Share Posted October 21, 2018 NVME is something we have considered but with the capacity we would need its prohibitive and there would be other bottlenecks. If we do use it - it will likely be as a cache for our regular SSD storage. Well be upgrading StorPool in the next week or two to employ hardware acceleration on our storage network which will reduce latency which is already at about a quarter millisecond. The processor upgrades are also improving storage speed and latency. Cached content performance should improve a little but not as much as everything else. My own personal site is on S1 and browsing the site without caching is very comparable to browsing with it at this point but thatll vary from site to site. At the end of the day if we had unlimited funding Im sure we could use the absolute latest and greatest of everything all the time but as we are small we do our best to continually invest in and upgrade our infrastructure, performance, and reliability. I think youll be happy with the increased performance even with caching. My WP-Admin is faster than Ive ever seen it and using WHM or cPanel is exceptionally fast. I dont know which server youre on off hand - but we will be upgrading all servers as soon as we can. Quote Link to comment Share on other sites More sharing options...
008Rohit Posted October 21, 2018 Report Share Posted October 21, 2018 Performance is indeed great at the moment. I'm on S1. When it comes to front-end cached pages, I was averaging around 650 ms prior to the processor upgrade, now it's around 500 ms. WP-admin feels a bit more responsive, too. As for NVMe, I had the chance to test the performance of a friend's site which is hosted on a server located in Germany which had NVMe SSDs, and it was indeed quite a bit faster there vs. the previous host. Avg. load time of cached pages went from around 450 ms to 220 ms for that site, with everything else being the same (LSCache enabled). But I also heard that NVMe servers cost up to 4-5x more in the US than in countries like Germany, and perhaps that's why I'm yet to hear about any US-based shared hosting server of any provider being powered by NVMe storage. About this processor upgrade, how big of an upgrade was it in terms of processor architecture? (i.e. Intel 4th gen. vs. 8th gen.) Clock speed hardly tells the whole story, as efficiency goes up every year. So, even though the newer processors are clocked 21% higher, I think the actual performance gain will be much more than that. Quote Link to comment Share on other sites More sharing options...
tshirtfly Posted October 21, 2018 Report Share Posted October 21, 2018 Hoping next would be P1 but i truly understand if it will take time. Great service from you guys. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 21, 2018 Author Report Share Posted October 21, 2018 Performance is indeed great at the moment. I'm on S1. When it comes to front-end cached pages, I was averaging around 650 ms prior to the processor upgrade, now it's around 500 ms. WP-admin feels a bit more responsive, too. As for NVMe, I had the chance to test the performance of a friend's site which is hosted on a server located in Germany which had NVMe SSDs, and it was indeed quite a bit faster there vs. the previous host. Avg. load time of cached pages went from around 450 ms to 220 ms for that site, with everything else being the same (LSCache enabled). But I also heard that NVMe servers cost up to 4-5x more in the US than in countries like Germany, and perhaps that's why I'm yet to hear about any US-based shared hosting server of any provider being powered by NVMe storage. About this processor upgrade, how big of an upgrade was it in terms of processor architecture? (i.e. Intel 4th gen. vs. 8th gen.) Clock speed hardly tells the whole story, as efficiency goes up every year. So, even though the newer processors are clocked 21% higher, I think the actual performance gain will be much more than that.They're the same generation - we're actually just going from low-power versions of the processors to higher power versions. 65 Watt TDP to 120 Watt TDP. It's something we've wanted to do for a long time but at ~$1,750 per processor it's not a cheap endeavor when you have 20+ to swap out. We thought it would just be the clock speed but the memory controller is handling ram faster - 2 Gigabytes/second faster as well. I've never seen MySQL on this server use so little CPU while doing more work and this isn't just because of the clock speed but because of how much faster RAM access is. Queries are running faster and more consistently. When it comes to NVMe - it's just like standard SSDs when we were all HDD - we're waiting on the price to come down to make it reasonable. Most already think we're expensive for what we give not really understanding how much it costs to provide the services we provide. Unfortunately shared hosting profit margins have gotten thinner and thinner over the years. It seems that people don't value quality support, reliability, and consistent speed like they used to and seem to value low price above anything else. We're just not willing to cram our servers as full as our competitors to improve our profit margins at the cost of performance and reliability. If we were to go to all NVMe - we'd most certainly have to double or triple pricing and the average hosting client isn't going to understand and would cancel/leave. If NVMe is something you want - we could certainly build you a custom solution but it would be substantially more expensive than anything else we could offer until NVMe comes down in price. Believe me - I'd love to be on NVMe and I remember the day when we were able to go pure SSD. There were days when we were on HDD that I'd just sit and dream about how nice it would be to be on SSD and not have to worry so much about IOPS and latency and throughput. Where things will 'just work' without having to be babysat so much . 1 Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 21, 2018 Author Report Share Posted October 21, 2018 Hoping next would be P1 but i truly understand if it will take time. Great service from you guys. P1 is slated for this week - and R1 is to follow. P1 will likely be Wednesday or Thursday and R1 will likely be Saturday or this next Monday. S1, R1, and P1 are our largest most populated servers from before our current architecture - they host 3 to 4 times the clients each that S2, R2, P2, etc host so once we get done with S1, R1, P1 we'll likely be able to do 2 or 3 servers at a time for upgrades. 1 Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 22, 2018 Author Report Share Posted October 22, 2018 We are swapping processors in one of our spare nodes now and will be moving P1 to it shortly. We might be able to upgrade another node today and move R1 to it - but that depends on the staffing at the facility. As this isn't an emergency we have to do it when the facility has normal staffing for normal maintenance. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 22, 2018 Author Report Share Posted October 22, 2018 It looks like we're going to have to move R1 and then P1 - but unless anything unexpected happens both will be getting upgraded today. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 22, 2018 Author Report Share Posted October 22, 2018 We're moving R1 now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 22, 2018 Author Report Share Posted October 22, 2018 R1 is moved. We're swapping processors to get ready to move P1. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 22, 2018 Author Report Share Posted October 22, 2018 We're going to move P1 now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 22, 2018 Author Report Share Posted October 22, 2018 P1 is back online [which also hosts this forum] . Quote Link to comment Share on other sites More sharing options...
tshirtfly Posted October 23, 2018 Report Share Posted October 23, 2018 I am getting better google speed rate. Is hardware acceleration enabled on p1 and r1 ? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 23, 2018 Author Report Share Posted October 23, 2018 Hardware acceleration is on the networking for our storage so it helps everything. S1, P1, R1 are on the upgraded processors so far. 1 Quote Link to comment Share on other sites More sharing options...
SarisIsop Posted October 24, 2018 Report Share Posted October 24, 2018 Yeah, I am definitely surprised at how large of an improvement this is. We will be upgrading all servers as quickly as we can. The original goal was to upgrade all servers at once but due to the major disaster and the damage it caused to our revenue we're having to do this in stages. http://www.screen-shot.net/2018-10-20_22-56-17.png This looks interesting, thank you. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 25, 2018 Author Report Share Posted October 25, 2018 It's definitely interesting. We're planning on having everybody upgraded - our whole network - within 30 days barring anything unexpected. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted January 12, 2019 Author Report Share Posted January 12, 2019 All processors in all compute nodes are fully upgraded . All shared, reseller, and VPS nodes will see the benefits of this upgrade. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.