Michael D. Posted February 27, 2014 Report Share Posted February 27, 2014 Hello, As I'm sure you know we use R1Soft to provide our incremental backups. We've found that R1Soft has been performing exceptionally slowly - backups that should take 15 to 30 minutes are taking 6 to 20 hours and, while the backup runs, server I/O and performance is severely degraded. R1Soft, unfortunately, has been causing slowness, unreliability, inconsistency, as well as short bouts of downtime. I really wish they would take this issue more seriously and work diligently on resolving it but it doesn't seem that it's a priority for them. I know of at least 5 other providers running R1Soft that all experience the same issues and are getting the same useless advice from R1Soft that clearly does not resolve the issue. Normally the I/O and performance degradation isn't an issue as we time our backups during off-peak times for our servers so that there is no impact to our customers, however, when the backup runs for 20+ hours - there is no 'off peak' time that this process will fit into. If you're curious - you can see the dismal transfer rates yourself:http://www.screen-shot.net/2014-02-27_11-53-36.png1.1 MB/sec average throughput and 194.1 KB/s current network raate and a total of ~8.5 hours to back up only **32 GB** of changed data over a GIGABIT network capable of 100 megabytes/second... http://www.screen-shot.net/2014-02-27_11-50-05.pngThis backup on another server says it's going to take ~28 hours for 89.9 GB of changed data and it shows the current rate at 2 KB/sec and the average throughput at 964 KB/s. By comparison here is what a fresh/seed backup looks like [could be faster, but it's not limited by the hardware or the network - but R1Soft itself]:http://www.screen-shot.net/2014-02-27_11-58-29.pngIt's clear that this seed backup transferred 238.2 GB over 6 hours 41 minutes at the average rate of 16.7 MB/s. While 16.7 MB/s isn't fast it's approximately 17 times faster than 964 KB/s or 1.1 MB/s. Here is another seed backup:http://www.screen-shot.net/2014-02-27_12-00-22.pngYou can see in this case 842.8 GB of data was transferred in 22 hours while an incremental backup of only 89 GB [second image from the top above] claims it will take 26+... We've been working with R1Soft to try and resolve this but so far have gotten nowhere. They requested we make sure we're running the absolutely latest version of the agents on the servers and we are and the performance on incremental backups continues to be dismal. For anybody that may think that the servers can't keep up or that the network simply isn't up to it, we'll go ahead and disprove these ahead of time. Network TestsBelow is the testing of the network between the backup server and a few other servers [ones that have been performing slowly on backups]. Each test is bi-directional testing sending and receiving: Atlantis VPS Node Backup Server Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.0.0.6 port 5001 connected with 10.0.0.11 port 44334 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec ------------------------------------------------------------ Client connecting to 10.0.0.11, TCP port 5001 TCP window size: 92.8 KByte (default) ------------------------------------------------------------ [ 4] local 10.0.0.6 port 52032 connected with 10.0.0.11 port 5001 [ 4] 0.0-10.0 sec 1.09 GBytes 939 Mbits/secKobold Server Backup Server ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.0.0.6 port 5001 connected with 10.0.0.13 port 47704 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec ------------------------------------------------------------ Client connecting to 10.0.0.13, TCP port 5001 TCP window size: 81.2 KByte (default) ------------------------------------------------------------ [ 4] local 10.0.0.6 port 40128 connected with 10.0.0.13 port 5001 [ 4] 0.0-10.0 sec 1.09 GBytes 939 Mbits/secJasmine Server Backup Server ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.0.0.6 port 5001 connected with 10.0.0.12 port 53892 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.01 GBytes 863 Mbits/sec ------------------------------------------------------------ Client connecting to 10.0.0.12, TCP port 5001 TCP window size: 135 KByte (default) ------------------------------------------------------------ [ 4] local 10.0.0.6 port 40944 connected with 10.0.0.12 port 5001 [ 4] 0.0-10.0 sec 1.01 GBytes 868 Mbits/secFile Read/Write [actual file data transfers]:Below you will see the results of us sending data from the same servers above to the backup server over the same network. Atlantis VPS Node --> Backup Server [root@atlantis ~]# scp 1gbfile.tgz root@priv.backup.mddhosting.com:/root root@priv.backup.mddhosting.com's password: 1gbfile.tgz 100% 1024MB 73.1MB/s 00:14Kobold Server --> Backup Server root@kobold [~]# scp 1gbfile.tgz root@priv.backup.mddhosting.com:/root 1gbfile.tgz 100% 1024MB 78.8MB/s 00:13Jasmine Server --> Backup Server root@echo [~]# scp 1gbfile.tgz root@priv.backup.mddhosting.com:/root 1gbfile.tgz 100% 1024MB 85.3MB/s 00:12The servers are clearly capable of reading and transmitting data much faster than R1Soft can perform. The bottleneck, in this case, is R1Soft and not the hardware or the network. We may have to look into alternatives to R1Soft in the interim or move down to performing backups weekly to keep from affecting performance. Quote Link to comment Share on other sites More sharing options...
gplurn Posted February 28, 2014 Report Share Posted February 28, 2014 Interesting, thanks for the info Mike. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted March 4, 2014 Author Report Share Posted March 4, 2014 R1Soft is going to be starting a backup on the Jasmine server here very shortly in their testing to diagnose the issue. I believe it will likely do a full block scan [takes ~24 hours] after which they will kick off another backup after a day to get an incremental backup for testing. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted March 12, 2014 Author Report Share Posted March 12, 2014 R1Soft is currently performing a seed backup on Kobold resulting in things being a little sluggish. It wasn't supposed to kick off a backup until Saturday but it looks like a setting was left unchanged mistakenly resulting in it running last night. By the time we noticed this it's about 75% done and it doesn't make much sense to stop it just to have to go through this again so we're going to let it run. Quote Link to comment Share on other sites More sharing options...
rubendob Posted April 6, 2016 Report Share Posted April 6, 2016 R1Soft is currently performing a seed backup on Kobold resulting in things being a little sluggish. It wasn't supposed to kick off a backup until Saturday but it looks like a setting was left unchanged mistakenly resulting in it running last night. By the time we noticed this it's about 75% done and it doesn't make much sense to stop it just to have to go through this again so we're going to let it run. Hi forgive me to bring back to top kind of very old thread but I'm curious...Were you able to fix this kind of poor performance? I'm in the same situation Quote Link to comment Share on other sites More sharing options...
Scott Posted April 6, 2016 Report Share Posted April 6, 2016 Hi forgive me to bring back to top kind of very old thread but I'm curious...Were you able to fix this kind of poor performance? I'm in the same situation We have a fix in the works, but I can't share any details just yet. Stay tuned. Quote Link to comment Share on other sites More sharing options...
rubendob Posted April 7, 2016 Report Share Posted April 7, 2016 We have a fix in the works, but I can't share any details just yet. Stay tuned. Hi I want to share my advances if you let me. I recreate all the Disk Safes for all servers (and attached to the right policies). The performance increase from 512 Kb/s to 30 MB/s. Is not problem with hardware in any case. Is something related with Disk Safes I bet. But it's very annoying to have to recreate the Disk Safes every X time... Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 1, 2016 Author Report Share Posted May 1, 2016 The older the disk safe [meaning the more recovery points it has had] the slower it gets. The disk safe itself gets heavily fragmented. You can 'defragment' them - but it takes so long on an old/slow disk safe that making new ones is the fastest/easiest way to resolve the issue. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.