Michael D. Posted August 3, 2013 Report Share Posted August 3, 2013 (edited) Hello. As a hosting provider that relies upon quality of service and word of mouth for growth it's obviously not in our best interests to have a server that's performing poorly or having issues. That said - we have a solid track record of keeping things fast, stable, and resolving issues quickly. The Icarus server, unfortunately, has been facing some issues as a result of the R1Soft backup system. This system, when operating normally, only copies the data changed from day to day - so while the server may have 500 GB of data stored, will only transmit the 3 to 5GB of data that actually *changed* from day to day. When there is an issue such as a power failure, a server reboot, or anything else that causes the backup agent on the server to get restarted - it has no way of knowing what has changed and what has not. In this sort of situation the backup server treats the server almost like it's new - it reads each and every block to determine what has or has not changed - it's a very intensive process that takes more time than a usual daily backup. Additionally if something goes wrong with the backup process - it can get 'stuck' and cause issues. For three days in a row we were diagnosing the backup system as it was indeed getting suck and causing issues. We believe we resolved the core issue causing this yesterday and had everything ready to go, however, our facility had a minor power outage to a part of the data center floor affecting both the backup server and the Icarus server. Due to this unexpected outage the backup server had to do another fresh backup last night. During this backup - the server was slower than usual. R1Soft is supposed to be able to be run on a server without adversely affecting it's performance but it seems this is not the case with the Icarus server. At the end of the day our newer hardware is more robust and handles load/backups better than our older hardware - and we're in the process of upgrading our hardware but we're not done with that yet. If this issue continues to persist with the Icarus server we'll be forced to migrate everybody off of it - however - we would like to do this as a part of our normal upgrade schedule. We will, however, perform migrations sooner rather than later if the backup system continues to cause issues for the Icarus server. We do deeply apologize for any issues the disruptions may have caused you. We do perform the backups during the least busy time for the server, however, that doesn't mean that the server is idle so it does still impact some customers. When faced with the option of having a couple of hours of slowness versus having no backups at all we choose to make sure we, at the least, have a safe backup copy of your data. Barring any unexpected issues - the server should perform a normal incremental backup tomorrow night and it should not cause issues. If you have any questions, by all means, feel free to ask. Edited August 7, 2013 by Scott S Marked resolved. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 3, 2013 Author Report Share Posted August 3, 2013 Only MySQL at this time appears to be running slow - we're investigating. I have issued a restart for MySQL just now which normally takes 5 to 10 seconds but is taking longer than expected. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 3, 2013 Author Report Share Posted August 3, 2013 The restart is still in progress - there is definitely something wrong with MySQL and we're going to continue to investigate. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 3, 2013 Author Report Share Posted August 3, 2013 MySQL is back online. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 3, 2013 Author Report Share Posted August 3, 2013 After the restart MySQL appears to be performing normally. I suspect some optimization needs to be performed - normally I would handle such optimizations but I will likely bring in a trusted administrator that is much more experienced than myself to give it a look due to the issues we've faced over the last few days. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 3, 2013 Author Report Share Posted August 3, 2013 I have made some optimizations on the server that should help prevent this issue from reoccurring. We're going to monitor the server, however, feel free to report if you feel anything is behaving abnormally. Quote Link to comment Share on other sites More sharing options...
Dean Posted August 4, 2013 Report Share Posted August 4, 2013 Hello Michael. MySQL is experiencing some issue on my websites. Thanks Quote Link to comment Share on other sites More sharing options...
Battra Posted August 4, 2013 Report Share Posted August 4, 2013 Right now I'm having problem accessing my sites. The server itself is up but it's either extremely slow or not loading at all. Quote Link to comment Share on other sites More sharing options...
100pr00f Posted August 4, 2013 Report Share Posted August 4, 2013 same thing here. My site has been almost down for about 3 days due to its slowness and my viewers and readers cant access the site at all. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 4, 2013 Author Report Share Posted August 4, 2013 Hey guys, Sorry I didn't catch this earlier - while the server performed better this morning it still was not up-to-par. I've adjusted the settings more and the issue shouldn't reoccur tomorrow. I did set up some special monitoring for this to wake me up if the issue reoccurred but it seems the site I chose to monitor wasn't intensive enough to be affected. The issue is only occurring when the R1Soft Backups [daily backups] are running. Recently some changes to R1Soft have caused it to be a little more intensive than usual resulting in slowness during the backup process. We do schedule the backups overnight to have the least impact but even during off-peak times we need our servers to perform well. The changes I've made will permit more data that needs written to the disks to be cached into RAM - this helps as during the backups the reads for the backups take priority and writes can be severely delayed resulting in slowness/downtime. I've set it substantially higher [upwards of 20% of system ram or 4.8 GB] which should be plenty to keep the server stable during backups. We do plan on replacing this hardware with much more robust hardware as soon as possible - likely within then next 6 months. The newer hardware can handle R1Soft backups mid-day even when the server is busy without anybody noticing. We're just a smaller company so we have to work towards that higher-end and every expensive hardware . Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 4, 2013 Author Report Share Posted August 4, 2013 It appears R1Soft may be causing issues with MySQL. For example: root@icarus [~]# mysqladmin statusmysqladmin: connect to server at 'localhost' failederror: 'User root already has more than 'max_user_connections' active connections' There is nothing going on now that would cause root to use more than a couple of connections. I'm restarting MySQL now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 4, 2013 Author Report Share Posted August 4, 2013 MySQL is back - if this issue occurs tomorrow [MySQL messing up] we'll have to disable the database-specific backups on this server. I'll set up an alarm tomorrow for me to check the server *as* the backup runs so I can see what is happening at the time of the failure. Quote Link to comment Share on other sites More sharing options...
100pr00f Posted August 4, 2013 Report Share Posted August 4, 2013 thanks Quote Link to comment Share on other sites More sharing options...
Scott Posted August 4, 2013 Report Share Posted August 4, 2013 same thing here. My site has been almost down for about 3 days due to its slowness and my viewers and readers cant access the site at all. If your site has been down all this time, the cause isn't the same as sites should be back to normal as soon as the backup stops or MySQL is restarted. Please open a support ticket so we can investigate your site specifically. Quote Link to comment Share on other sites More sharing options...
kothet Posted August 5, 2013 Report Share Posted August 5, 2013 Here is what happened to me. Yesterday was 5th day that I had this problem. Started since 31st July. I opened the ticket in day 1, 2, 3 and 4 and I don't bother opening ticket yesterday anymore. Everyday, at the same time, the site went down or being very very slow. Support didn't even notice they had problem until I opened ticket in Day 1 and 2. In one ticket, they even said site is fine and probably my ISP. When I said my clients had same problem from other countries, then they started to send ticket to level 2 support, who took about 3 hours in day 2 and even longer in day 4. Even my free monitor system alerted me about this problem right away and you guys didn't seems to notice until I opened the ticket. You said you will try to migrate the sites to another server when it happened again. It did happened again yesterday and you still have no solution. This is too much for mddhosting for a company who guarantee 99/9% uptime. Although you don't even mention about compensating your valued customers, I do have to compensate my value clients who I care about. 5 straight days with same problem and hours to fix the same issue? I am moving all my accounts to another reliable host. My mistake about MDD hosting. BTW, I can be patience up to 3 days. Not 5 days. If today happened, it will be 6th. Quote Link to comment Share on other sites More sharing options...
Scott Posted August 5, 2013 Report Share Posted August 5, 2013 Backups have started on the icarus server as of a few minutes ago. Everything appears stable, but we will continue to monitor the server until backups have completed. Quote Link to comment Share on other sites More sharing options...
kothet Posted August 5, 2013 Report Share Posted August 5, 2013 its happening again i think. Loading slow and my monitors are sending me alerts... Quote Link to comment Share on other sites More sharing options...
luan152 Posted August 5, 2013 Report Share Posted August 5, 2013 Yep, my websites are slow, too. Quote Link to comment Share on other sites More sharing options...
Battra Posted August 5, 2013 Report Share Posted August 5, 2013 Any update on this issue? My sites are not responding again ... Quote Link to comment Share on other sites More sharing options...
kothet Posted August 5, 2013 Report Share Posted August 5, 2013 6 days straight. Almost 2 hours since the problem started today and no solution yet. I am moving my client sites to another host since this morning and I think i made the right decision. Quote Link to comment Share on other sites More sharing options...
Dean Posted August 5, 2013 Report Share Posted August 5, 2013 Morning Guys, Same issue again, Now i am starting to get angry customers Any Updates? Quote Link to comment Share on other sites More sharing options...
Dehenderson Posted August 5, 2013 Report Share Posted August 5, 2013 My sites, all running WordPress, have ground to a halt. It's not possible to load the wp-admin of WordPress. I need to be making updates online and cannot because of this protracted problem with the Icarus server. Update ... now, my sites are down, giving me "Error establishing a database connection." Quote Link to comment Share on other sites More sharing options...
Dean Posted August 5, 2013 Report Share Posted August 5, 2013 What happened to 24/7 support? Quote Link to comment Share on other sites More sharing options...
Scott Posted August 5, 2013 Report Share Posted August 5, 2013 Hello Everyone, I'd like to apologize for our lack of responsivness in the forums and in tickets this morning. As you know, we are a small team, and this sometimes means we have to make a hard choice between spending our time working on an issue or responding the deluge of tickets reporting the issue. Today we opted to focus our efforts in stablizing and debugging the database issue on icarus. At this time, the MySQL server is functioning normally and your sites should be as well. We were forced to terminate the MySQL process, so you may see a moment or two of a specific database not responding as the mysqlcheck program reviews and repairs any potential corruption due to the unclean shutdown. I don't have anything new to report in terms of a cause or resolution to the problem, however we did gather extensive logs prior to shutting down the MySQL process. It will take us some time to review these, but it should give us an insight into what happened when MySQL locked up. We will post another update when we have new information. In the meantime, we ask that any issues be reported via support ticket and not here on the forum. We need to keep announcement threads clean so anyone who is trying to get up to speed on the issue can easily and clearly a record of our announcements. Quote Link to comment Share on other sites More sharing options...
100pr00f Posted August 5, 2013 Report Share Posted August 5, 2013 My site is running slow but is accessible for others to view all day so far today. (not like the last 4 days where you cant even load the site in the mornings for about 6 to 8 hours) But it is sitll slow. only status that has a warning sign which has been like that for about 5 days in the cpanel is Disk /dev/sda5 (/) 82 % sorry to hear others still having a harder time with there site. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.