Jump to content
MDDHosting Forums

Kobold Server Reboot [Required due to R1Soft Failure] - 12:01 AM ET 02/11/2014 - ~10 minutes


Recommended Posts

Unfortunately the Idera Hot Copy [HCP] is 'stuck' on the Kobold server and cannot be stopped:

... thousands of lines similar to the next ...
Mar 10 20:08:15 kobold kernel: [14648846.361057] hcp: INFO: Shutdown request 15997 on hcp device 1 with -1 users and 1 cdp users.
Mar 10 20:08:15 kobold kernel: [14648846.367053] hcp: INFO: Shutdown request 15998 on hcp device 1 with -1 users and 1 cdp users.
Mar 10 20:08:15 kobold kernel: [14648846.373034] hcp: INFO: Shutdown request 15999 on hcp device 1 with -1 users and 1 cdp users.
Mar 10 20:08:15 kobold kernel: [14648846.379050] hcp: INFO: Shutdown request 16000 on hcp device 1 with -1 users and 1 cdp users.
Mar 10 20:08:15 kobold kernel: [14648846.385032] hcp: ERROR: Can not stop mounted hcp session hcp1, please unmount and try again.
kobold [~]# hcp -l
Idera Hot Copy     5.4.4 build 95 (http://www.r1soft.com)
Documentation      http://wiki.r1soft.com
Forums             http://forum.r1soft.com

Thank you for using Hot Copy!
Idera makes the only Continuous Data Protection software for Linux.

****** hcp1 ******
 Real Device:           /dev/sda3
 Virtual Device:        /dev/hcp1
 Changed Blocks Stored: /.r1soft_hcp_sda3.cow_hcp1
 Mounted:               -
 Time Created:          Sun Mar 09 22:30:02 EDT 2014
 Changed Blocks:        71925.81 MiB (75419680768 bytes)

We got with R1Soft and, unfortunately, the only solution to this problem is to reboot the server. We made it clear to them that this isn't acceptable but that doesn't change the fact that the reboot is required to fix it. Until we reboot the server disk I/O performance is going to suffer [hence why we're scheduling this on such short notice].

 

After the reboot we will be installing a version that allegedly fixes/resolves this issue [i.e. prevents it from happening again] according to R1Soft support.

 

The total downtime should be no more than 10 minutes barring a file system check or anything unexpected on the reboot. We will update this thread when the server is being shut down and booted back up as well as any udpates for anything unexpected.

Link to comment
Share on other sites

The server is taking a bit longer to stabilize - I believe due to the number of firewall rules that had to be processed on boot [due to the recent WordPress brute force attacks.

 

We've made some changes that should alleviate this issue in the future should we have to reboot.

Link to comment
Share on other sites

The system is still loading ~4,000 firewall rules - we're going to trim those down now that the attacks have largely passed once it's done loading them. Things should stabilize very quickly once they're done loading and it's happening at the rate of a few per second.

Link to comment
Share on other sites

The latest CloudLinux kernel is reporting exceptionally high load [although Idle CPU and disk I/O are perfect]:

top - 00:46:07 up 39 min,  2 users,  load average: 212.36, 198.92, 123.88
Tasks: 1409 total,   8 running, 1401 sleeping,   0 stopped,   0 zombie
Cpu(s): 16.5%us,  8.1%sy,  0.0%ni, 73.9%id,  1.0%wa,  0.0%hi,  0.5%si,  0.0%st

We may have to perform another restart in the very near future but the next one will go much more smoothly due to us pruning the firewall rules substantially.

 

I have opened a ticket with CloudLinux support - this should not cause any immediate issues although statistics processing may be delayed until this is resolved.

Link to comment
Share on other sites

A level 3 administrator was able to sort the issue out. It looks like MySQL didn't quite start up properly - it was going super slow at handling queries causing them to stack up and the load to skyrocket. I can't say I understand everything the admin told me to be honest - but the good news is the issue is resolved without a reboot :).

 

Going to mark this as resolved.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...