Jump to content


Photo

R1Soft - Dismal Incremental Backup Speeds resulting in Severe Degredation of Server Performance


  • Please log in to reply
7 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 27 February 2014 - 12:35 PM

Hello,

 

As I'm sure you know we use R1Soft to provide our incremental backups.  We've found that R1Soft has been performing exceptionally slowly - backups that should take 15 to 30 minutes are taking 6 to 20 hours and, while the backup runs, server I/O and performance is severely degraded.

 

R1Soft, unfortunately, has been causing slowness, unreliability, inconsistency, as well as short bouts of downtime.  I really wish they would take this issue more seriously and work diligently on resolving it but it doesn't seem that it's a priority for them.  I know of at least 5 other providers running R1Soft that all experience the same issues and are getting the same useless advice from R1Soft that clearly does not resolve the issue.

 

Normally the I/O and performance degradation isn't an issue as we time our backups during off-peak times for our servers so that there is no impact to our customers, however, when the backup runs for 20+ hours - there is no 'off peak' time that this process will fit into.

 

If you're curious - you can see the dismal transfer rates yourself:

2014-02-27_11-53-36.png

1.1 MB/sec average throughput and 194.1 KB/s current network raate and a total of ~8.5 hours to back up only **32 GB** of changed data over a GIGABIT network capable of 100 megabytes/second...

 

2014-02-27_11-50-05.png

This backup on another server says it's going to take ~28 hours for 89.9 GB of changed data and it shows the current rate at 2 KB/sec and the average throughput at 964 KB/s.

 

By comparison here is what a fresh/seed backup looks like [could be faster, but it's not limited by the hardware or the network - but R1Soft itself]:

2014-02-27_11-58-29.png

It's clear that this seed backup transferred 238.2 GB over 6 hours 41 minutes at the average rate of 16.7 MB/s.  While 16.7 MB/s isn't fast it's approximately 17 times faster than 964 KB/s or 1.1 MB/s.

 

Here is another seed backup:

2014-02-27_12-00-22.png

You can see in this case 842.8 GB of data was transferred in 22 hours while an incremental backup of only 89 GB [second image from the top above] claims it will take 26+...

 

We've been working with R1Soft to try and resolve this but so far have gotten nowhere.  They requested we make sure we're running the absolutely latest version of the agents on the servers and we are and the performance on incremental backups continues to be dismal.

 

For anybody that may think that the servers can't keep up or that the network simply isn't up to it, we'll go ahead and disprove these ahead of time.

 

Network Tests

Below is the testing of the network between the backup server and a few other servers [ones that have been performing slowly on backups].  Each test is bi-directional testing sending and receiving:

 

Atlantis VPS Node <--> Backup Server

Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.0.0.6 port 5001 connected with 10.0.0.11 port 44334
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec
------------------------------------------------------------
Client connecting to 10.0.0.11, TCP port 5001
TCP window size: 92.8 KByte (default)
------------------------------------------------------------
[  4] local 10.0.0.6 port 52032 connected with 10.0.0.11 port 5001
[  4]  0.0-10.0 sec  1.09 GBytes   939 Mbits/sec

Kobold Server <--> Backup Server

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.0.0.6 port 5001 connected with 10.0.0.13 port 47704
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec
------------------------------------------------------------
Client connecting to 10.0.0.13, TCP port 5001
TCP window size: 81.2 KByte (default)
------------------------------------------------------------
[  4] local 10.0.0.6 port 40128 connected with 10.0.0.13 port 5001
[  4]  0.0-10.0 sec  1.09 GBytes   939 Mbits/sec

Jasmine Server <--> Backup Server

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.0.0.6 port 5001 connected with 10.0.0.12 port 53892
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.01 GBytes   863 Mbits/sec
------------------------------------------------------------
Client connecting to 10.0.0.12, TCP port 5001
TCP window size:  135 KByte (default)
------------------------------------------------------------
[  4] local 10.0.0.6 port 40944 connected with 10.0.0.12 port 5001
[  4]  0.0-10.0 sec  1.01 GBytes   868 Mbits/sec

File Read/Write [actual file data transfers]:

Below you will see the results of us sending data from the same servers above to the backup server over the same network.

 

Atlantis VPS Node --> Backup Server

[root@atlantis ~]# scp 1gbfile.tgz root@priv.backup.mddhosting.com:/root
root@priv.backup.mddhosting.com's password:
1gbfile.tgz        100% 1024MB  73.1MB/s   00:14

Kobold Server --> Backup Server

root@kobold [~]# scp 1gbfile.tgz root@priv.backup.mddhosting.com:/root
1gbfile.tgz        100% 1024MB  78.8MB/s   00:13

Jasmine Server --> Backup Server

root@echo [~]# scp 1gbfile.tgz root@priv.backup.mddhosting.com:/root
1gbfile.tgz        100% 1024MB  85.3MB/s   00:12

The servers are clearly capable of reading and transmitting data much faster than R1Soft can perform.  The bottleneck, in this case, is R1Soft and not the hardware or the network.

 

We may have to look into alternatives to R1Soft in the interim or move down to performing backups weekly to keep from affecting performance.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#2 gplurn

gplurn

    Newbie

  • Members
  • Pip
  • 12 posts

Posted 28 February 2014 - 10:47 AM

Interesting, thanks for the info Mike.


  • 0

#3 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 04 March 2014 - 06:39 PM

R1Soft is going to be starting a backup on the Jasmine server here very shortly in their testing to diagnose the issue.  I believe it will likely do a full block scan [takes ~24 hours] after which they will kick off another backup after a day to get an incremental backup for testing.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#4 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 12 March 2014 - 01:14 PM

R1Soft is currently performing a seed backup on Kobold resulting in things being a little sluggish.  It wasn't supposed to kick off a backup until Saturday but it looks like a setting was left unchanged mistakenly resulting in it running last night.  By the time we noticed this it's about 75% done and it doesn't make much sense to stop it just to have to go through this again so we're going to let it run.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#5 rubendob

rubendob

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 06 April 2016 - 03:55 AM

R1Soft is currently performing a seed backup on Kobold resulting in things being a little sluggish.  It wasn't supposed to kick off a backup until Saturday but it looks like a setting was left unchanged mistakenly resulting in it running last night.  By the time we noticed this it's about 75% done and it doesn't make much sense to stop it just to have to go through this again so we're going to let it run.

 

Hi 

 

forgive me to bring back to top kind of very old thread but I'm curious...Were you able to fix this kind of poor performance? I'm in the same situation :(


  • 0

#6 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 06 April 2016 - 05:35 PM

 

Hi 

 

forgive me to bring back to top kind of very old thread but I'm curious...Were you able to fix this kind of poor performance? I'm in the same situation :(

 

We have a fix in the works, but I can't share any details just yet. Stay tuned.


  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#7 rubendob

rubendob

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 07 April 2016 - 04:06 AM

 

We have a fix in the works, but I can't share any details just yet. Stay tuned.

 

Hi

 

I want to share my advances if you let me. I recreate all the Disk Safes for all servers (and attached to the right policies). The performance increase from 512 Kb/s to 30 MB/s. Is not problem with hardware in any case. Is something related with Disk Safes I bet. 

 

 

But it's very annoying to have to recreate the Disk Safes every X time...


  • 0

#8 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 01 May 2016 - 01:00 AM

The older the disk safe [meaning the more recovery points it has had] the slower it gets.  The disk safe itself gets heavily fragmented.

 

You can 'defragment' them - but it takes so long on an old/slow disk safe that making new ones is the fastest/easiest way to resolve the issue.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users