Michael D. Posted May 9, 2023 Report Share Posted May 9, 2023 We are working to restore service to the R4 server at this time. We believe our virtualization manager may have attempted to boot the server on two machines at once resulting in the file system going read-only. We might need to perform a file system check or we may need to roll back to a snapshot from a few minutes prior to the issue. Service will be restored as quickly as possible - we don't want downtime any more than you do. Sorry for the trouble this is causing. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 9, 2023 Author Report Share Posted May 9, 2023 The server is booting back up now. Quote Link to comment Share on other sites More sharing options...
BASEdriven Posted May 9, 2023 Report Share Posted May 9, 2023 Thanks for all your work on this! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 9, 2023 Author Report Share Posted May 9, 2023 The system is performing file system checks on boot, we will keep you updated here. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted May 9, 2023 Author Report Share Posted May 9, 2023 The R4 server is now back online. However, due to data corruption, we had to roll back the server to our most recent snapshot of the storage, resulting in a loss of approximately 11 minutes of new data. We attempted to recover the live data initially, but the process would have taken significantly longer and likely caused hours of additional downtime. Given the minimal data loss due to our snapshot frequency, we decided that rolling back a few minutes was a better alternative to prolonged downtime. As some of you may be aware, the R4 server has experienced stability issues in the past week. We had planned to rebuild the machine configuration and migrate to it tonight in order to avoid further problems. However, the issue we were trying to prevent occurred this morning before that could be scheduled and executed. As a result, we have already rebuilt the machine onto a fresh new configuration, eliminating the need for tonight's scheduled maintenance. If you have any questions or concerns, please let us know. Feel free to reach out either here, directly through a support ticket, or via live chat. We will be performing a deep investigation into this issue and providing a Reason For Outage (RFO) within 72 hours. 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.