Jump to content
MDDHosting Forums

Icarus Slowness / Issues Recently

Recommended Posts



As a hosting provider that relies upon quality of service and word of mouth for growth it's obviously not in our best interests to have a server that's performing poorly or having issues. That said - we have a solid track record of keeping things fast, stable, and resolving issues quickly.


The Icarus server, unfortunately, has been facing some issues as a result of the R1Soft backup system. This system, when operating normally, only copies the data changed from day to day - so while the server may have 500 GB of data stored, will only transmit the 3 to 5GB of data that actually *changed* from day to day.


When there is an issue such as a power failure, a server reboot, or anything else that causes the backup agent on the server to get restarted - it has no way of knowing what has changed and what has not. In this sort of situation the backup server treats the server almost like it's new - it reads each and every block to determine what has or has not changed - it's a very intensive process that takes more time than a usual daily backup.


Additionally if something goes wrong with the backup process - it can get 'stuck' and cause issues. For three days in a row we were diagnosing the backup system as it was indeed getting suck and causing issues. We believe we resolved the core issue causing this yesterday and had everything ready to go, however, our facility had a minor power outage to a part of the data center floor affecting both the backup server and the Icarus server. Due to this unexpected outage the backup server had to do another fresh backup last night.


During this backup - the server was slower than usual. R1Soft is supposed to be able to be run on a server without adversely affecting it's performance but it seems this is not the case with the Icarus server. At the end of the day our newer hardware is more robust and handles load/backups better than our older hardware - and we're in the process of upgrading our hardware but we're not done with that yet.


If this issue continues to persist with the Icarus server we'll be forced to migrate everybody off of it - however - we would like to do this as a part of our normal upgrade schedule. We will, however, perform migrations sooner rather than later if the backup system continues to cause issues for the Icarus server.


We do deeply apologize for any issues the disruptions may have caused you. We do perform the backups during the least busy time for the server, however, that doesn't mean that the server is idle so it does still impact some customers. When faced with the option of having a couple of hours of slowness versus having no backups at all we choose to make sure we, at the least, have a safe backup copy of your data.


Barring any unexpected issues - the server should perform a normal incremental backup tomorrow night and it should not cause issues.


If you have any questions, by all means, feel free to ask.

Edited by Scott S
Marked resolved.
Link to comment
Share on other sites

After the restart MySQL appears to be performing normally. I suspect some optimization needs to be performed - normally I would handle such optimizations but I will likely bring in a trusted administrator that is much more experienced than myself to give it a look due to the issues we've faced over the last few days.

Link to comment
Share on other sites

Hey guys,


Sorry I didn't catch this earlier - while the server performed better this morning it still was not up-to-par. I've adjusted the settings more and the issue shouldn't reoccur tomorrow. I did set up some special monitoring for this to wake me up if the issue reoccurred but it seems the site I chose to monitor wasn't intensive enough to be affected.


The issue is only occurring when the R1Soft Backups [daily backups] are running. Recently some changes to R1Soft have caused it to be a little more intensive than usual resulting in slowness during the backup process. We do schedule the backups overnight to have the least impact but even during off-peak times we need our servers to perform well.


The changes I've made will permit more data that needs written to the disks to be cached into RAM - this helps as during the backups the reads for the backups take priority and writes can be severely delayed resulting in slowness/downtime. I've set it substantially higher [upwards of 20% of system ram or 4.8 GB] which should be plenty to keep the server stable during backups.


We do plan on replacing this hardware with much more robust hardware as soon as possible - likely within then next 6 months. The newer hardware can handle R1Soft backups mid-day even when the server is busy without anybody noticing. We're just a smaller company so we have to work towards that higher-end and every expensive hardware :).

Link to comment
Share on other sites

It appears R1Soft may be causing issues with MySQL.


For example:


root@icarus [~]# mysqladmin status
mysqladmin: connect to server at 'localhost' failed
error: 'User root already has more than 'max_user_connections' active connections'


There is nothing going on now that would cause root to use more than a couple of connections. I'm restarting MySQL now.

Link to comment
Share on other sites

MySQL is back - if this issue occurs tomorrow [MySQL messing up] we'll have to disable the database-specific backups on this server. I'll set up an alarm tomorrow for me to check the server *as* the backup runs so I can see what is happening at the time of the failure.

Link to comment
Share on other sites

same thing here. My site has been almost down for about 3 days due to its slowness and my viewers and readers cant access the site at all.


If your site has been down all this time, the cause isn't the same as sites should be back to normal as soon as the backup stops or MySQL is restarted. Please open a support ticket so we can investigate your site specifically.

Link to comment
Share on other sites

Here is what happened to me.


Yesterday was 5th day that I had this problem. Started since 31st July. I opened the ticket in day 1, 2, 3 and 4 and I don't bother opening ticket yesterday anymore.


Everyday, at the same time, the site went down or being very very slow. Support didn't even notice they had problem until I opened ticket in Day 1 and 2. In one ticket, they even said site is fine and probably my ISP. When I said my clients had same problem from other countries, then they started to send ticket to level 2 support, who took about 3 hours in day 2 and even longer in day 4.


Even my free monitor system alerted me about this problem right away and you guys didn't seems to notice until I opened the ticket. You said you will try to migrate the sites to another server when it happened again. It did happened again yesterday and you still have no solution.


This is too much for mddhosting for a company who guarantee 99/9% uptime. Although you don't even mention about compensating your valued customers, I do have to compensate my value clients who I care about.


5 straight days with same problem and hours to fix the same issue? I am moving all my accounts to another reliable host. My mistake about MDD hosting.


BTW, I can be patience up to 3 days. Not 5 days. If today happened, it will be 6th.

Link to comment
Share on other sites

My sites, all running WordPress, have ground to a halt. It's not possible to load the wp-admin of WordPress. I need to be making updates online and cannot because of this protracted problem with the Icarus server.


Update ... now, my sites are down, giving me "Error establishing a database connection."

Link to comment
Share on other sites

Hello Everyone,


I'd like to apologize for our lack of responsivness in the forums and in tickets this morning. As you know, we are a small team, and this sometimes means we have to make a hard choice between spending our time working on an issue or responding the deluge of tickets reporting the issue. Today we opted to focus our efforts in stablizing and debugging the database issue on icarus. At this time, the MySQL server is functioning normally and your sites should be as well. We were forced to terminate the MySQL process, so you may see a moment or two of a specific database not responding as the mysqlcheck program reviews and repairs any potential corruption due to the unclean shutdown. I don't have anything new to report in terms of a cause or resolution to the problem, however we did gather extensive logs prior to shutting down the MySQL process. It will take us some time to review these, but it should give us an insight into what happened when MySQL locked up.


We will post another update when we have new information. In the meantime, we ask that any issues be reported via support ticket and not here on the forum. We need to keep announcement threads clean so anyone who is trying to get up to speed on the issue can easily and clearly a record of our announcements.

Link to comment
Share on other sites

My site is running slow but is accessible for others to view all day so far today. (not like the last 4 days where you cant even load the site in the mornings for about 6 to 8 hours) But it is sitll slow.


only status that has a warning sign which has been like that for about 5 days in the cpanel is


Disk /dev/sda5 (/) 82 %


sorry to hear others still having a harder time with there site.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...