
[Completed] Fresh Backups of Echo / Cypress / Fresco
#21
Posted 04 April 2011 - 01:32 PM
#22
Posted 04 April 2011 - 01:33 PM
1. Let it finish and have accurate verified backups of the server and your data.
2. Cancel it now, after it's been running so long, have it run tomorrow and cause issues again tomorrow.
3. Cancel the backup and then there is no backup should there be hardware failure or another unexpected form of data loss.
We worked to prioritize the web server and MySQL and de-prioritize the backup system but it seems that there is still an underlying disk performance issue at work that is causing problems. Try as hard as we can to get things to stabilize some sites are operating normally while others are simply not operating well or at all.
We do estimate once the backup verification finishes that things should speed up and get back to normal however if not we're going to have to continue investigating to see if there is another underlying issue causing performance degradation. The backup verification itself should have finished in less than 10 hours however it has been running for 15 hours and reports 17 minutes remaining.
We're actually going to be making big changes to the semi-dedicated platform as we're not happy with the current server's ability to keep up with intensive disk situations such as long-term backups.
I'm also working on bringing online a VPS temporarily where I can stage any accounts that are under 10 GB of disk usage that wish to be moved off of Cypress until the changes to the semi-dedicated offerings are completed.
I'm going to be posting a very detailed message as to why Cypress has been facing issues and what we plan on doing to fix the issue permanently. All customers on Cypress will be getting an email with the information as well as a link to the thread I'm going to post once it's ready.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#23
Posted 04 April 2011 - 01:44 PM
Brian, sending you a PM as I'm very interested to learn more about Cloudflare.
#24
Posted 04 April 2011 - 01:44 PM
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#25
Posted 04 April 2011 - 01:48 PM
I would love a temporary VPS until we can find a permanent fix -- assuming it's fully managed. My Linux skills are C+.Indeed the server is having some issues right now - it looks like R1Soft decided to verify it's full backup from yesterday which I've never seen it do before. R1Soft has been less than helpful but ultimately we have three choices:
1. Let it finish and have accurate verified backups of the server and your data.
2. Cancel it now, after it's been running so long, have it run tomorrow and cause issues again tomorrow.
3. Cancel the backup and then there is no backup should there be hardware failure or another unexpected form of data loss.
We worked to prioritize the web server and MySQL and de-prioritize the backup system but it seems that there is still an underlying disk performance issue at work that is causing problems. Try as hard as we can to get things to stabilize some sites are operating normally while others are simply not operating well or at all.
We do estimate once the backup verification finishes that things should speed up and get back to normal however if not we're going to have to continue investigating to see if there is another underlying issue causing performance degradation. The backup verification itself should have finished in less than 10 hours however it has been running for 15 hours and reports 17 minutes remaining.
We're actually going to be making big changes to the semi-dedicated platform as we're not happy with the current server's ability to keep up with intensive disk situations such as long-term backups.
I'm also working on bringing online a VPS temporarily where I can stage any accounts that are under 10 GB of disk usage that wish to be moved off of Cypress until the changes to the semi-dedicated offerings are completed.
I'm going to be posting a very detailed message as to why Cypress has been facing issues and what we plan on doing to fix the issue permanently. All customers on Cypress will be getting an email with the information as well as a link to the thread I'm going to post once it's ready.
#26
Posted 04 April 2011 - 02:21 PM
The server is now done and *should* perform well now. It may take a few minutes for things to get back to normal.I would love a temporary VPS until we can find a permanent fix -- assuming it's fully managed. My Linux skills are C+.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#27
Posted 04 April 2011 - 02:22 PM
I appreciate your efforts and you always seem to be working on it right away, however it seems to have endless issues.
#28
Posted 04 April 2011 - 02:28 PM
#29
Posted 04 April 2011 - 02:59 PM
I'd be interested in a temporary, fully managed VPS as well since Cypress is currently not working properly about once a week on average.
I appreciate your efforts and you always seem to be working on it right away, however it seems to have endless issues.
If you can wait until I get my open letter that I am writing done and then make your decisions based upon that message, I would appreciate it. I am going to list some options and detail some changes that we will be making over the next couple of days, weeks, and months.What would be involved in getting a temp vps? Would there be downtime?
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
#30
Posted 06 April 2011 - 12:41 PM
R1Soft appears to be the real problem here, and is causing significant cost increases for the luxury of daily backups. Just isn't worth it IMHO

#31
Posted 06 April 2011 - 01:30 PM
Weekly backups (although I'd add daily SQL backups, since thats whats most likely to change) doesn't seem like a bad idea in theory though. Although the server seemed to melt that day he installed the SQL backup plugin too, so who knows.
I don't know why it's so hard to have something scanning files in the background at a sane rate and copying anything with changes, but apparently it's too much for R1Soft.
Maybe their name is short for Remedial comp.sci. 1 Software

Hopefully whoever maintains cloud linux can get the version with the IO improvements fixed, seemed like the server load dropped by half the couple of days that was on.
#32
Posted 06 April 2011 - 01:31 PM
#33
Posted 06 April 2011 - 01:59 PM
R1Soft probably ain't cheap as well. Maybe, if you can, develop something in-house.
That would most likely cost a lot more.
#34
Posted 07 April 2011 - 05:15 AM
You're welcome to stay on the existing account if you don't want to make a change to one of the new plans or a VPS at 50% off. The issue with R1Soft 3.0 is that if you stop a backup in progress, you reboot, or you restart the CDP agent on the server - it will do a full scan on the next run no matter what. I've discussed this with one of the individuals at R1Soft pivotal in the development of their CDP agent and they're going to be addressing this in a few various ways that will all make this process much better but it will take time (on the order of months) to get it built and rolled out on R1's side.Mike, have you considered switching to weekly only backups on Cypress? I personally wouldn't have a problem with that, as daily backups for large sized accounts seems rather overkill anyway.
R1Soft appears to be the real problem here, and is causing significant cost increases for the luxury of daily backups. Just isn't worth it IMHO
In discussing the details with a source at R1Soft we've figured out why the full backups are happening but it has been an unfortunate series of domino events.I think the issue with that is that would always require full backups, which really pound the server for a long time vs the daily "supposed to only backup changes" backups. From what I understand it's just been a bunch of unfortunate events that have been causing it to do full backups more often recently.
It's just the way the 3.0 MySQL module is written - IMHO it's TERRIBLE. It spends 3 to 10 seconds PER DATABASE when the 2.0 version could back up all MySQL databases on the entire server in seconds if not a couple of minutes. The 3.0 version would take HOURS upon HOURS every night just to back up the MySQL databases.Weekly backups (although I'd add daily SQL backups, since thats whats most likely to change) doesn't seem like a bad idea in theory though. Although the server seemed to melt that day he installed the SQL backup plugin too, so who knows.
They're actually adding throttling and some other really neat things that should help a lot but they won't be available, again, for months.I don't know why it's so hard to have something scanning files in the background at a sane rate and copying anything with changes, but apparently it's too much for R1Soft.
Maybe their name is short for Remedial comp.sci. 1 Software
They have a new version out that is supposedly stable but we're going to give it 2 to 3 weeks before we make the switch so others using it can find these "unexpected system crashes", report them, and have them resolved first.Hopefully whoever maintains cloud linux can get the version with the IO improvements fixed, seemed like the server load dropped by half the couple of days that was on.
It's not cheap, that's for sure at $500 or so per server last time I looked. Developing software to do this in-house would certainly be much more expensive however for sure. We considered simply using rsync backups but our servers tend to have millions upon millions of files which means that rsync would spend a tremendous amount of time just identifying what files to copy which is fairly intensive in and of itself.R1Soft probably ain't cheap as well. Maybe, if you can, develop something in-house.
Indeed.That would most likely cost a lot more.
█ Scalable shared hosting plans in the cloud! Check them out!
█ Highly Available Cloud Shared, Reseller, and VPS
█ http://www.mddhosting.com/
1 user(s) are reading this topic
0 members, 1 guests, 0 anonymous users