As a hosting provider that relies upon quality of service and word of mouth for growth it's obviously not in our best interests to have a server that's performing poorly or having issues. That said - we have a solid track record of keeping things fast, stable, and resolving issues quickly.
The Icarus server, unfortunately, has been facing some issues as a result of the R1Soft backup system. This system, when operating normally, only copies the data changed from day to day - so while the server may have 500 GB of data stored, will only transmit the 3 to 5GB of data that actually *changed* from day to day.
When there is an issue such as a power failure, a server reboot, or anything else that causes the backup agent on the server to get restarted - it has no way of knowing what has changed and what has not. In this sort of situation the backup server treats the server almost like it's new - it reads each and every block to determine what has or has not changed - it's a very intensive process that takes more time than a usual daily backup.
Additionally if something goes wrong with the backup process - it can get 'stuck' and cause issues. For three days in a row we were diagnosing the backup system as it was indeed getting suck and causing issues. We believe we resolved the core issue causing this yesterday and had everything ready to go, however, our facility had a minor power outage to a part of the data center floor affecting both the backup server and the Icarus server. Due to this unexpected outage the backup server had to do another fresh backup last night.
During this backup - the server was slower than usual. R1Soft is supposed to be able to be run on a server without adversely affecting it's performance but it seems this is not the case with the Icarus server. At the end of the day our newer hardware is more robust and handles load/backups better than our older hardware - and we're in the process of upgrading our hardware but we're not done with that yet.
If this issue continues to persist with the Icarus server we'll be forced to migrate everybody off of it - however - we would like to do this as a part of our normal upgrade schedule. We will, however, perform migrations sooner rather than later if the backup system continues to cause issues for the Icarus server.
We do deeply apologize for any issues the disruptions may have caused you. We do perform the backups during the least busy time for the server, however, that doesn't mean that the server is idle so it does still impact some customers. When faced with the option of having a couple of hours of slowness versus having no backups at all we choose to make sure we, at the least, have a safe backup copy of your data.
Barring any unexpected issues - the server should perform a normal incremental backup tomorrow night and it should not cause issues.
If you have any questions, by all means, feel free to ask.
Edited by Scott S, 07 August 2013 - 09:09 AM.