Tim Posted February 16, 2017 Report Share Posted February 16, 2017 Hello, We are aware of the issues with the R2 server and are awaiting MySQL to shut down so we can reboot the server. I will update this thread once we have more information. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 16, 2017 Report Share Posted February 16, 2017 It looks like our backup software, JetBackup, exhausted all of the available free disk space on the server.While we do have monitoring and alerting for disk space so that we can take steps prior to an outage - in this case the space was used up so quickly [well over 150GB of space used in minutes] that we were unable to rectify the issue prior to the server running completely out. Once the server exhausted the disk space it became unresponsive to the point that we waited about 10 minutes for it to attempt to reboot on it's own. It was restarting but it was taking an exorbitant amount of time so we kicked off a hard reset. The server came back up, we cleared the extraneous space by the backup system, and restarted the server again to give it a fresh start. As of this moment MySQL is starting up and the server should be fully available within a few minutes.We are working on moving to a new storage platform that allows more flexibility - one that would have avoided this issue entirely - but it's not quite ready yet.It will take several minutes for it to return to normal. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 16, 2017 Report Share Posted February 16, 2017 The MySQL server on R2 is presently recovering from the InnoDB crash that resulted from the initial forced restart. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 16, 2017 Report Share Posted February 16, 2017 MySQL is back online. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 16, 2017 Report Share Posted February 16, 2017 Everything is back online but MySQL is repairing any tables that may have been getting written to when the restart happened. 1 Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 16, 2017 Report Share Posted February 16, 2017 The server is stable at this point - we are slowly re-enabling some of the ancillary services to keep the load down and the server as responsive as possible. If you are still experiencing any issues at all do please reach out to technical support. We are sorry for the trouble this has caused. We've taken potential issues such as this into consideration when looking at our new storage platform. Ideally we'll be moving to the new platform in the very near future. 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.