Michael D. Posted October 15, 2015 Report Share Posted October 15, 2015 Hello, The Echo server went unresponsive unexpectedly at 10:14 AM ET. We were unable to connect remotely to the server in any way including to our remote KVM solution. The server was forcefully reset and upon coming back online now requires a file system check. This check can take anwaywhere from 30 minutes to 4 hours. Ideally it will be closer to 30 minutes, however, I have no way of knowing exactly how long it will take. I will do my best to keep you update as to the progress of the check as well as restoration of services. We will have this back online as quickly as possible - we don't enjoy downtime any more than you do. If you have any general questions about this outage post them here - any specific issues about your account do please open a support ticket. Understand we're likely going to be experiencing a heavier ticket load than normal so responses may be a little delayed but we will do our best to answer your questions as quickly as possible. We are sorry for the trouble this is causing you and will restore services as quickly as we can. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 15, 2015 Author Report Share Posted October 15, 2015 80% Completed on the first pass. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 15, 2015 Author Report Share Posted October 15, 2015 http://www.screen-shot.net/2015-10-15_10-54-28.png Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 15, 2015 Author Report Share Posted October 15, 2015 85.1% Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 15, 2015 Author Report Share Posted October 15, 2015 87.5% Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 15, 2015 Author Report Share Posted October 15, 2015 The server is booting up now. It is possible there may still be some issues for us to resolve such as crashed databases, etc. If your site is experiencing issues after this outage do please open a ticket and be as detailed as possible. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 15, 2015 Author Report Share Posted October 15, 2015 The server is coming back online. As it is one of our older servers it's going to take it a little while to 'catch up' and normalize. We are planning on upgrading this server within the next 2 to 6 weeks [we're waiting on the hardware] to much more robust and powerful hardware. No pricing changes either . Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 15, 2015 Author Report Share Posted October 15, 2015 The server is reporting online now - again - it may be sluggish for a little while as caches build and it catches up to missed requests. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 15, 2015 Author Report Share Posted October 15, 2015 The MySQL server is doing InnoDB Crash Recovery. I wish I could provide an ETA but it does not provide one and I would just be guessing. As soon as crash recovery is done MySQL will be back online. Everything else is online - email, FTP, cPanel, the web server itself. Anything not relying on MySQL will function. Anything requiring MySQL will function once the crash recovery is completed. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 15, 2015 Author Report Share Posted October 15, 2015 MySQL is back online and now the server is really being hammered. It will stabilize but until it does it will be slow/extremely slow. We are going to be rolling out some tech to this server prior to the new hardware upgrades that should help with this as well but it's just not quite ready. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted October 15, 2015 Author Report Share Posted October 15, 2015 Sever load is coming down - it still has a ways to go but it's getting there. Quote Link to comment Share on other sites More sharing options...
Tim Posted October 15, 2015 Report Share Posted October 15, 2015 The server is now back online and should be back to normal shortly. (If not already.) Please open a support ticket if you continue to have any issues. Quote Link to comment Share on other sites More sharing options...
AverageForumUser Posted October 15, 2015 Report Share Posted October 15, 2015 Thank you very much Mike for keeping us updated, always appreciated. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.