Michael D. Posted April 4, 2011 Report Share Posted April 4, 2011 Important Update 04/17/2011LiteSpeed Technologies, the company who produces our web server software, has released a new revision of their software that incorporates a feature that dramatically increases the web server speed and reliability especially under heavy load conditions such as during a backup. Please read this thread for full details:http://forums.mddhosting.com/topic/483-litespeed-41-released-aio-means-extremely-increased-reliability/ ----------------------- This is a message that was emailed to all customers on our semi-dedicated platform including the Cypress server concerning recent performance and reliability issues we have faced on the platform as well as where we are taking the platform in the future. 1. Where we are now, what has happened, and why. I am sending you this message, as well as posting it on our public forums, so that you can understand why we have been having the issues with the server your account is located on over the past week. It has always been my policy to be as open and honest with our customers and not to try and hide issues, spin issues, or otherwise provide any information that is less than accurate. Mistakes have definitely been made by us on several levels when it comes to our semi-dedicated offerings and these mistakes have caused the platform to be less reliable and to lack the performance that the platform is designed to have. I am going to go into detail as to what mistakes we have made, why those mistakes were made, and what plans we're putting in place to bring semi-dedicated back to it's planned levels of performance and reliability. I do understand that some of this may be too technical for some of you so if you need any part of this message clarified or explained in more detail, please do not hesitate to respond and let me know what you need clarified. Originally we started offering services on Intel Quad Core servers with 4 GB of DDR2 RAM and RAID10 with four 250GB SATA2 hard disks. On these servers we were able to put around 250 accounts while maintaining high reliability and performance across the board for all of our clients on these servers. For a long time shared hosting is all we offered and the plans on these servers worked well for almost everybody without issues however there were a few customers who were running very heavy scripts and wanted more CPU and Disk I/O available for them to use as they needed. Upon putting great thought into the needs of this special set of customers we came up with our new semi-dedicated offerings. Our semi-dedicated servers were originally the exact same configuration as our regular shared servers and the only difference was that instead of having up to 250 accounts they were originally limited to only 25 accounts. Running the same hardware with fewer accounts required higher price points for the accounts. With such a tight limit on the accounts we were able to take the total disk and bandwidth allocations for the server itself and evenly divide them up for our clients to use. This was an excellent platform for our customers who needed more CPU, RAM, and Disk space than our average customers. Over time as we've worked to improve our services we've upgraded server hardware and even moved to fully owned hardware instead of leasing our equipment. There have been several hardware configurations between the original quad core servers and the current dual quad core servers with hyper-threading that all accounts are now served from. The servers we use now are extremely powerful and robust compared to the equipment that we originally started offering services on which has meant changes for all of our plans over time. One large change for all plans was the move from SATA2 7.2k RPM drives to SAS 15k RPM drives. The SATA2 drives we had been running were enterprise class drives however there are many benefits to SAS drives in a busy server environment. The first major difference between the drives is that the maximum data capacity of SAS drives is substantially lower than the maximum capacity of a SATA2 drive. Currently the largest SAS 15k RPM drive you can get is 600 GB and costs on the order of $500 each where as the largest SATA2 7.2k RPM drive you can get is on the order of 2,000 GB at $220. The second major difference between a SAS drive and a SATA drive is the number of IOPS the drives can do, meaning the number of input and output operations per second. In a shared environment with a lot of random reading/writing such as a web server the higher the IOPS, the better the performance. When we made the move from leased hardware to owned hardware we were facing an interesting situation on our semi-dedicated offerings that we had not faced with any other plans or hardware in the past. On our semi-dedicated servers we had plenty of CPU and RAM to go around however we were running out of disk space. The average regular shared customer uses under 1 GB of disk space and roughly less than half a percent of regular shared customers use anywhere near their actual disk allocations. We found over time that the opposite was true of semi-dedicated offerings and that the average semi-dedicated customer will use 34 GB of disk space and only roughly one half of a percent will use less than 1 GB of disk space. Our first major mistake was increasing the disk allocations to semi-dedicated customers when we increased the disk allocations for all other plans including shared, reseller, and VPS offerings. We increased these allocations before we learned that the average semi-dedicated customer will use most of what they're given and, as they say, hindsight is 20/20. Our second major mistake when we went back to the higher capacity SATA2 disks to give the semi-dedicated servers the capacity they needed to service the accounts instead of admitting our first mistake and rectifying the situation by reducing allocations and making other changes to the accounts. The primary reason we opted for going for the larger and slower disks instead of reducing the disk space allocated to the accounts was simply due to the trouble it would have caused our customers at the time and the fact that we believed that the SATA2 disks could keep up due to the extremely limited number of accounts and this is true 99% of the time. On average when there is not a backup being processed on the server or anything other system-wide that is disk intensive the semi-dedicated servers hum along beautifully and perform just as we had expected and intended. As you likely know we've had performance and reliability issues with the server that you are on over about the last week and those reliability issues stem from two different yet related issues. The first issue is that we had upgraded to a newer version of the Operating System on the server as the new version promised improved disk performance and reliability and this was one of the major issues we had faced with the server your account is located on due to the slower SATA2 disks. It turns out that the version we had installed which was supposed to be stable was actually not as stable as expected and caused the server to crash every few nights at random times and almost always seemed to interrupt the R1Soft backup system that was running on the server to protect your data. R1Soft requires a full server scan which can take up to 15 hours to perform as it backs up every bit of data on the server the first time that it runs. Each time the system runs after that point it is supposed to only look for data that has changed and then copy that data over which means that the process only takes between 30 and 120 minutes each night and tends not to inhibit performance. One major downside to R1Soft that has resulted in the issues the server has faced is that anytime a backup process is interrupted R1Soft will perform a full server scan again on the next cycle instead of checking for changed data as it should. We feel that keeping up to date backups of the server and your data is of the utmost importance as a majority of hosting customers tend not to perform their own regular backups and, as such, we've run these backups although they have caused performance issues. We do use R1Soft as needed to restore files and databases for our customers when they make a mistake such as dropping the wrong database or they have other issues where they don't have their own backups and their site is damaged. We faced an issue last week where a database was not able to be restored via it's raw files on the disk to the server which resulted in the user having lost about 3 years worth of data in their database. Now understand this correctly - we didn't lose their data - the customer did be it a mistake on their part or a mistake in the script they were working with however we did do what we could to help them get their data back to no avail. Having faced this issue with a database not being able to be restored via the raw files on the disk we ordered the MySQL add-on for R1Soft and activated it for the server your account is on. This module is supposed to make a complete MySQL replication of every database allowing us to restore down to the individual table level as needed instead of simply having to shotgun replace the entire database in the event that there is an issue with a database. This new module did not perform as expected and instead of taking a few minutes to back up the databases it actually lengthened the backup process each night from about an hour out to about 15 to 20 hours during which time the server was sluggish and nearly unresponsive. This is originally what caused issues at the start of this week. After determining that the MySQL addon wasn't going to do what we needed to do and that it was causing more issues than it helped us solve we canceled the backup process and things returned to normal. As you know from my explanation above, R1Soft will perform an intensive full block scan on the server the next time the backup process is run which resulted in more performance and response issues while the backup system performed this process once again. As of today we have full backups of the server that your account is on and the system will now go back to performing the short and less intensive backups of the server that should not degrade performance and should run less than an hour every night. This is good news but it's not the end of this message and we will be making changes to the semi-dedicated platform in the near future to eliminate these issues entirely and permanently. 2. Where we are going, what changes will be made, and why. Having admitted the mistakes we've made we're going to now tell you what we plan on doing with our semi-dedicated platform in the future to remedy these mistakes. It's not going to be an easy transition for those that wish to go through with it however we're not going to force anybody to make the changes. If you're currently happy with your plan and service you are welcome to keep things as-is on the current server. Now that we have a full backup of the server and we're on a stable version of the operating system we plan on leaving things as-is for as long as possible to avoid the performance issues you have faced over the last week. We are going to be bringing online a new server for semi-dedicated accounts using the same high speed SAS 15k RPM disk drives that we use in our shared, reseller, and VPS servers. What this means is that a full backup scan on the new server will only take 4 to 6 hours and that during this time performance will be impacted minimally. Due to the high speed of the SAS drives even while performing a full block scan the server should still be very responsive and the difference in load times should be on the order of milliseconds instead of seconds. While performance, speed, and reliability will be increased by going to these high speed SAS drives the total storage capacity will be greatly reduced. Our current SATA2 powered semi-dedicated servers can house up to 2,700 GB of data where as our new server will max out at around 576 GB of data. The resulting major decrease in disk capacity means that we will be forced to greatly reduce the disk allocations of all semi-dedicated level plans. These plans are expected to range from 2 GB to 5 GB versus the current 25 GB to 100 GB while the pricing is expected to remain the same. With disk allocations of 2 GB to 5 GB per account we can allow each and every customer to use 100% of their disk allocations and still place these customers on high speed SAS 15k RPM drives. Anybody who is currently on a semi-dedicated plan now would be welcome to make the transfer to the new server however the account will be required to conform to the new disk limits on the new server. Anybody who wishes to keep their current allocations will be welcome to stay on the existing server for as long as they keep their current account active. There is a possibility that we may be able to acquire the newer 600GB SAS15k RPM drives which would effectively double the capacity of the server as well as allowing us to double the allocations to 4 GB to 10 GB per account however we will need to have accounting run the numbers before we can determine if this is feasible. Another option for existing semi-dedicated customers only is that we will provide you a Basic VPS at a recurring 50% discount. The number of VPS we can provision with this offer is limited so if this is something that you wish to take advantage of I would suggest doing so as soon as possible. One thing that I do wish to make clear before anybody says that even at 50% the VPS is too expensive is that we actually pay $10 per month for cPanel licensing per VPS which brings the maximum income we see per account would be $17.50 which is actually lower than the cheapest semi-dedicated account we're currently offering. Our goal in moving you to a VPS is not to make more money off of you but to improve the reliability and performance for you without greatly increasing your cost. I do hope that you will get back to me with any of your thoughts, suggestions, opinions, etc... We do expect things to be stable on the server you are on for quite some time so if you do wish to wait a bit to see how things work out, that is up to you. Thank you, Quote Link to comment Share on other sites More sharing options...
username Posted April 4, 2011 Report Share Posted April 4, 2011 Out of sheer curiosity, have you looked into SSD's at all for the new server? They still suck in terms of $/GB compared to regular drives, but not so much vs server drives. As far as IOPS they blow every spinning disk type drive on the planet out of the water so you wouldn't need any RAID for striping, only redundancy (although since they tend to fail differently than regular drives might not even need that with daily backups), which might reduce the number of disks required - but I don't know if it would be enough to get the costs in line.If you haven't looked it might be worth while, could even be a bit of a selling point. If you have and decided against them, fair enough. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted April 4, 2011 Author Report Share Posted April 4, 2011 Out of sheer curiosity, have you looked into SSD's at all for the new server? They still suck in terms of $/GB compared to regular drives, but not so much vs server drives. As far as IOPS they blow every spinning disk type drive on the planet out of the water so you wouldn't need any RAID for striping, only redundancy (although since they tend to fail differently than regular drives might not even need that with daily backups), which might reduce the number of disks required - but I don't know if it would be enough to get the costs in line.If you haven't looked it might be worth while, could even be a bit of a selling point. If you have and decided against them, fair enough.We have, and we've actually been looking at offering a hosting service based off of SSDs however their prices are still extremely expensive and we're waiting for them to come down a bit before we can invest into them. We honestly have no issues with our SAS 15k RPM disk powered servers and expect them to treat us well up until SSD becomes feasible from a financial standpoint. Quote Link to comment Share on other sites More sharing options...
Adam Posted April 4, 2011 Report Share Posted April 4, 2011 Maybe you guys should look into putting the databases on SSDs, if you can figure out a way to do this. It would speed up response time a lot. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted April 4, 2011 Author Report Share Posted April 4, 2011 Maybe you guys should look into putting the databases on SSDs, if you can figure out a way to do this. It would speed up response time a lot.That is something that we've considered but we'd never run a single SSD (always at least two in Raid1) and we'd need to acquire two very expensive SSD drives as well as an additional raid controller per server to make it happen. It's actually more cost-effective to just build out another server. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted April 4, 2011 Author Report Share Posted April 4, 2011 Another thing we are considering is going with an 8 port raid controller and 4 300gb SAS 15k RPM disks to start with and then once those are getting full tossing an additional 4 disks into the server and creating another raid10 array. This would effectively allow us to double the capacity and I/O without having to build out another server. We'll be weighing all of our options between now and the time that the new server is actually ordered. Quote Link to comment Share on other sites More sharing options...
Blind Bandit Posted April 5, 2011 Report Share Posted April 5, 2011 Another thing we are considering is going with an 8 port raid controller and 4 300gb SAS 15k RPM disks to start with and then once those are getting full tossing an additional 4 disks into the server and creating another raid10 array. This would effectively allow us to double the capacity and I/O without having to build out another server. We'll be weighing all of our options between now and the time that the new server is actually ordered. I was just going to say going with an 8 drive array would allow for more space. Such as going with a raid array of 4 or 8 600gig SAS drives. Granted the price might need to go up. But really if someone needs 10 to 20 gigs of storage the price might need to go up according. And if you 30 plus your probably looking at VPS territory. I was thinking maybe its time to bump up semi dedicated pricing to 40-60 from the current 20-30. Its currently a steal and really if people need a lot of resources and a lot of space they are going to have to pay for it. It seems like a waste to only offering a few gigs of space for people who need a lot of resources. I would argue that 40-60 dollars is still a good deal if that can include 10-20 gigs of space or at least more than 5. That might also allow for a lower server populations. That would allowing keeping the older sever up for those that didn't want to move. And not short new members or put MDD in the poor house for eating excessive costs. I don't know if a lot of people know but MDDs pricing for semi dedicated is lower than most offerings out there. Granted some offer lower sever populations but you sure do pay for it. last time I checked it wasn't out of the norm to see 100 bucks plus for a very large semi dedicated plan. That I think puts things in prospective a bit. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted April 5, 2011 Author Report Share Posted April 5, 2011 We're definitely evaluating our options for new customers. I can't say that pricing won't go up for new customers however we're generally in the business of not raising prices for existing customers. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted April 6, 2011 Author Report Share Posted April 6, 2011 The new plans and their pricing are now live at http://www.mddhosting.com/semidedicated.php. Any orders placed between now and the time that the new server is online will be placed on a VPS that offers comparable performance to the actual hardware that will be rolled out. Since the pricing is a bit different than existing pricing - we will work with any customers on the old semi-dedicated plans that wish to make a shift to the new plans. Quote Link to comment Share on other sites More sharing options...
username Posted April 6, 2011 Report Share Posted April 6, 2011 As long as any current accounts on old plans are staying as is, maybe this is getting off topic a bit - but you might want to make it more obvious on your info pages why someone would get one of the top two semi-dedi plans at the new prices vs a vps. For $5 more you get more space and BW. For the top end you can get a vps for $25 less with more space and BW, not to mention what appears to be a much larger share of ram and cpu and overall server. That's your sales staff's problem not mine though, so feel free to disregard this, I'm sure you've got lots of other things to do with all the new servers coming online soon. Quote Link to comment Share on other sites More sharing options...
stocktrader Posted April 6, 2011 Report Share Posted April 6, 2011 you might want to make it more obvious on your info pages why someone would get one of the top two semi-dedi plans at the new prices vs a vps. A more broader page which explains the plans a bit more and answers questions like "Which plan is right for me?" would be better. It could address issues like: - "I am running a forum, what do I need?"- "My site gets 500 visitors a day, is a shared plan good enough?" , etc This would also be effective as a pre-sell page for pure beginners. Many beginners know nothing about web hosting. When I first started my site, I actually had to post on a USENET group asking what the word "bandwidth" meant. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted April 6, 2011 Author Report Share Posted April 6, 2011 As long as any current accounts on old plans are staying as is, maybe this is getting off topic a bit - but you might want to make it more obvious on your info pages why someone would get one of the top two semi-dedi plans at the new prices vs a vps. For $5 more you get more space and BW. For the top end you can get a vps for $25 less with more space and BW, not to mention what appears to be a much larger share of ram and cpu and overall server. That's your sales staff's problem not mine though, so feel free to disregard this, I'm sure you've got lots of other things to do with all the new servers coming online soon.The semi-dedicated plans will be targeting an entirely different market than those who would be looking for a VPS. You would likely be the type to choose the VPS over the semi-dedicated and that's ultimately your decision to make. A more broader page which explains the plans a bit more and answers questions like "Which plan is right for me?" would be better. It could address issues like: - "I am running a forum, what do I need?"- "My site gets 500 visitors a day, is a shared plan good enough?" , etc This would also be effective as a pre-sell page for pure beginners. Many beginners know nothing about web hosting. When I first started my site, I actually had to post on a USENET group asking what the word "bandwidth" meant.We are planning on fully revamping the website in the next 6 months or so with a totally new design at which point all content and marketing on the site will be reviewed and updated as necessary to be more in-line with our current offerings. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.