Jump to content
MDDHosting Forums

R2 Unplanned Outage - 02/16/17 - 7:30 AM - 8:45 AM ET


Recommended Posts

It looks like our backup software, JetBackup, exhausted all of the available free disk space on the server.

While we do have monitoring and alerting for disk space so that we can take steps prior to an outage - in this case the space was used up so quickly [well over 150GB of space used in minutes] that we were unable to rectify the issue prior to the server running completely out.


Once the server exhausted the disk space it became unresponsive to the point that we waited about 10 minutes for it to attempt to reboot on it's own. It was restarting but it was taking an exorbitant amount of time so we kicked off a hard reset.


The server came back up, we cleared the extraneous space by the backup system, and restarted the server again to give it a fresh start.


As of this moment MySQL is starting up and the server should be fully available within a few minutes.

We are working on moving to a new storage platform that allows more flexibility - one that would have avoided this issue entirely - but it's not quite ready yet.

It will take several minutes for it to return to normal.

Link to comment
Share on other sites

The server is stable at this point - we are slowly re-enabling some of the ancillary services to keep the load down and the server as responsive as possible.


If you are still experiencing any issues at all do please reach out to technical support.


We are sorry for the trouble this has caused. We've taken potential issues such as this into consideration when looking at our new storage platform. Ideally we'll be moving to the new platform in the very near future.

  • Upvote 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...