Actively Monitored Demeter Server -> Possible short outage this evening [60~120 seconds].

[Michael D.]

Hello!

The software we use to back up our servers has been causing problems for the Demeter server - resulting in crashes and downtime at night. We turned off R1Soft backups for Demeter on Friday to ensure the server remained online and operational.

 

We are re-enabling these backups tonight and are starting a manual backup so that we can actively watch the server and watch for the problem. Thus far we've only been able to catch the issue after the server went unresponsive and not prior making our investigation difficult.

 

We do expect the server to crash at some point this evening, however, a reboot takes approximately 20 seconds and we are going to be actively watching the server and not simply relying on our internal and external monitoring. If the server does crash we will bring it back online within 60 to 120 seconds - if not - we'll update this thread.

 

We've exhausted our options with our software vendor for investigating this and need to see the fault in action.

 

The downtime should be brief enough that it largely goes unnoticed. We did still want to give some heads-up so that if it does crash our customers are not wondering why.

 

Ultimately it's incredibly important that we be able to keep recent backups of client data for disaster recovery and, as such, it's important that we resolve this issue.

 

If you have any questions or concerns about this at all - let us know.

[ericr]

The backup has completed without any issue. I will be doing another manual backup tomorrow and will update this post prior to starting.

[ericr]

I am starting today's backup and monitoring it's progress.