Jump to content


Photo

R2 Unplanned Outage - 02/16/17 - 7:30 AM - 8:45 AM ET

R2

  • Please log in to reply
5 replies to this topic

#1 Tim

Tim

    Member

  • Staff
  • PipPip
  • 60 posts
  • Gender:Male

Posted 16 February 2017 - 08:14 AM

Hello,

 

We are aware of the issues with the R2 server and are awaiting MySQL to shut down so we can reboot the server. I will update this thread once we have more information.


  • 0

#2 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,631 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 16 February 2017 - 08:34 AM

It looks like our backup software, JetBackup, exhausted all of the available free disk space on the server.

While we do have monitoring and alerting for disk space so that we can take steps prior to an outage - in this case the space was used up so quickly [well over 150GB of space used in minutes] that we were unable to rectify the issue prior to the server running completely out.

 

Once the server exhausted the disk space it became unresponsive to the point that we waited about 10 minutes for it to attempt to reboot on it's own.  It was restarting but it was taking an exorbitant amount of time so we kicked off a hard reset.

 

The server came back up, we cleared the extraneous space by the backup system, and restarted the server again to give it a fresh start.

 

As of this moment MySQL is starting up and the server should be fully available within a few minutes.

We are working on moving to a new storage platform that allows more flexibility - one that would have avoided this issue entirely - but it's not quite ready yet.

It will take several minutes for it to return to normal.


  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!


#3 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,631 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 16 February 2017 - 08:38 AM

The MySQL server on R2 is presently recovering from the InnoDB crash that resulted from the initial forced restart.


  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!


#4 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,631 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 16 February 2017 - 08:40 AM

MySQL is back online.


  • 0

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!


#5 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,631 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 16 February 2017 - 08:49 AM

Everything is back online but MySQL is repairing any tables that may have been getting written to when the restart happened.


  • 1

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!


#6 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,631 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 16 February 2017 - 09:26 AM

The server is stable at this point - we are slowly re-enabling some of the ancillary services to keep the load down and the server as responsive as possible.

 

If you are still experiencing any issues at all do please reach out to technical support.

 

We are sorry for the trouble this has caused.  We've taken potential issues such as this into consideration when looking at our new storage platform.  Ideally we'll be moving to the new platform in the very near future.


  • 1

Michael Denney - MDDHosting, LLC - Professional Hosting Solutions
LiteSpeed Powered - Shared, Reseller, Premium, and VPS
Incremental R1Soft CDP Backups on all shared, semi-dedicated, and VPS services!
http://www.mddhosting.com/- Follow us on Twitter!






1 user(s) are reading this topic

0 members, 1 guests, 0 anonymous users