33 replies to this topic

[Lincoln]

I heard from Michael who has been working on the issue, which should be straightened out very soon. It had to do with setting the priority of the processes, which emphasized the web server and SQL but not PHP processes.

[MikeDVB]

Indeed the server is having some issues right now - it looks like R1Soft decided to verify it's full backup from yesterday which I've never seen it do before. R1Soft has been less than helpful but ultimately we have three choices:
1. Let it finish and have accurate verified backups of the server and your data.
2. Cancel it now, after it's been running so long, have it run tomorrow and cause issues again tomorrow.
3. Cancel the backup and then there is no backup should there be hardware failure or another unexpected form of data loss.

We worked to prioritize the web server and MySQL and de-prioritize the backup system but it seems that there is still an underlying disk performance issue at work that is causing problems. Try as hard as we can to get things to stabilize some sites are operating normally while others are simply not operating well or at all.

We do estimate once the backup verification finishes that things should speed up and get back to normal however if not we're going to have to continue investigating to see if there is another underlying issue causing performance degradation. The backup verification itself should have finished in less than 10 hours however it has been running for 15 hours and reports 17 minutes remaining.

We're actually going to be making big changes to the semi-dedicated platform as we're not happy with the current server's ability to keep up with intensive disk situations such as long-term backups.

I'm also working on bringing online a VPS temporarily where I can stage any accounts that are under 10 GB of disk usage that wish to be moved off of Cypress until the changes to the semi-dedicated offerings are completed.

I'm going to be posting a very detailed message as to why Cypress has been facing issues and what we plan on doing to fix the issue permanently. All customers on Cypress will be getting an email with the information as well as a link to the thread I'm going to post once it's ready.

[Lincoln]

Glad you're on top of this Michael, sounds like a typical day for me when dealing with my Windows 95 computer at work. Lord.

Brian, sending you a PM as I'm very interested to learn more about Cloudflare.

[MikeDVB]

I'm drafting an open letter now, and everybody should have a copy (and it will be posted on the forums) within an hour or so, maybe a little longer.

[Brian Stevenson]

Indeed the server is having some issues right now - it looks like R1Soft decided to verify it's full backup from yesterday which I've never seen it do before. R1Soft has been less than helpful but ultimately we have three choices:
1. Let it finish and have accurate verified backups of the server and your data.
2. Cancel it now, after it's been running so long, have it run tomorrow and cause issues again tomorrow.
3. Cancel the backup and then there is no backup should there be hardware failure or another unexpected form of data loss.

We worked to prioritize the web server and MySQL and de-prioritize the backup system but it seems that there is still an underlying disk performance issue at work that is causing problems. Try as hard as we can to get things to stabilize some sites are operating normally while others are simply not operating well or at all.

We do estimate once the backup verification finishes that things should speed up and get back to normal however if not we're going to have to continue investigating to see if there is another underlying issue causing performance degradation. The backup verification itself should have finished in less than 10 hours however it has been running for 15 hours and reports 17 minutes remaining.

We're actually going to be making big changes to the semi-dedicated platform as we're not happy with the current server's ability to keep up with intensive disk situations such as long-term backups.

I'm also working on bringing online a VPS temporarily where I can stage any accounts that are under 10 GB of disk usage that wish to be moved off of Cypress until the changes to the semi-dedicated offerings are completed.

I'm going to be posting a very detailed message as to why Cypress has been facing issues and what we plan on doing to fix the issue permanently. All customers on Cypress will be getting an email with the information as well as a link to the thread I'm going to post once it's ready.

I would love a temporary VPS until we can find a permanent fix -- assuming it's fully managed. My Linux skills are C+.

[MikeDVB]

I would love a temporary VPS until we can find a permanent fix -- assuming it's fully managed. My Linux skills are C+.

The server is now done and *should* perform well now. It may take a few minutes for things to get back to normal.

[le.gentleman]

I'd be interested in a temporary, fully managed VPS as well since Cypress is currently not working properly about once a week on average.
I appreciate your efforts and you always seem to be working on it right away, however it seems to have endless issues.

[Ilan]

What would be involved in getting a temp vps? Would there be downtime?

[MikeDVB]

I'd be interested in a temporary, fully managed VPS as well since Cypress is currently not working properly about once a week on average.
I appreciate your efforts and you always seem to be working on it right away, however it seems to have endless issues.

What would be involved in getting a temp vps? Would there be downtime?

If you can wait until I get my open letter that I am writing done and then make your decisions based upon that message, I would appreciate it. I am going to list some options and detail some changes that we will be making over the next couple of days, weeks, and months.

[TotalZen]

Mike, have you considered switching to weekly only backups on Cypress? I personally wouldn't have a problem with that, as daily backups for large sized accounts seems rather overkill anyway.

R1Soft appears to be the real problem here, and is causing significant cost increases for the luxury of daily backups. Just isn't worth it IMHO

[username]

I think the issue with that is that would always require full backups, which really pound the server for a long time vs the daily "supposed to only backup changes" backups. From what I understand it's just been a bunch of unfortunate events that have been causing it to do full backups more often recently.

Weekly backups (although I'd add daily SQL backups, since thats whats most likely to change) doesn't seem like a bad idea in theory though. Although the server seemed to melt that day he installed the SQL backup plugin too, so who knows.

I don't know why it's so hard to have something scanning files in the background at a sane rate and copying anything with changes, but apparently it's too much for R1Soft.
Maybe their name is short for Remedial comp.sci. 1 Software

Hopefully whoever maintains cloud linux can get the version with the IO improvements fixed, seemed like the server load dropped by half the couple of days that was on.

[Ilan]

R1Soft probably ain't cheap as well. Maybe, if you can, develop something in-house.

[Blind Bandit]

R1Soft probably ain't cheap as well. Maybe, if you can, develop something in-house.

That would most likely cost a lot more.

[MikeDVB]

Mike, have you considered switching to weekly only backups on Cypress? I personally wouldn't have a problem with that, as daily backups for large sized accounts seems rather overkill anyway.

R1Soft appears to be the real problem here, and is causing significant cost increases for the luxury of daily backups. Just isn't worth it IMHO

You're welcome to stay on the existing account if you don't want to make a change to one of the new plans or a VPS at 50% off. The issue with R1Soft 3.0 is that if you stop a backup in progress, you reboot, or you restart the CDP agent on the server - it will do a full scan on the next run no matter what. I've discussed this with one of the individuals at R1Soft pivotal in the development of their CDP agent and they're going to be addressing this in a few various ways that will all make this process much better but it will take time (on the order of months) to get it built and rolled out on R1's side.

I think the issue with that is that would always require full backups, which really pound the server for a long time vs the daily "supposed to only backup changes" backups. From what I understand it's just been a bunch of unfortunate events that have been causing it to do full backups more often recently.

In discussing the details with a source at R1Soft we've figured out why the full backups are happening but it has been an unfortunate series of domino events.

Weekly backups (although I'd add daily SQL backups, since thats whats most likely to change) doesn't seem like a bad idea in theory though. Although the server seemed to melt that day he installed the SQL backup plugin too, so who knows.

It's just the way the 3.0 MySQL module is written - IMHO it's TERRIBLE. It spends 3 to 10 seconds PER DATABASE when the 2.0 version could back up all MySQL databases on the entire server in seconds if not a couple of minutes. The 3.0 version would take HOURS upon HOURS every night just to back up the MySQL databases.

I don't know why it's so hard to have something scanning files in the background at a sane rate and copying anything with changes, but apparently it's too much for R1Soft.
Maybe their name is short for Remedial comp.sci. 1 Software

They're actually adding throttling and some other really neat things that should help a lot but they won't be available, again, for months.

Hopefully whoever maintains cloud linux can get the version with the IO improvements fixed, seemed like the server load dropped by half the couple of days that was on.

They have a new version out that is supposedly stable but we're going to give it 2 to 3 weeks before we make the switch so others using it can find these "unexpected system crashes", report them, and have them resolved first.

R1Soft probably ain't cheap as well. Maybe, if you can, develop something in-house.

It's not cheap, that's for sure at $500 or so per server last time I looked. Developing software to do this in-house would certainly be much more expensive however for sure. We considered simply using rsync backups but our servers tend to have millions upon millions of files which means that rsync would spend a tremendous amount of time just identifying what files to copy which is fairly intensive in and of itself.

That would most likely cost a lot more.

Indeed.