Michael D. Posted February 26, 2014 Report Share Posted February 26, 2014 Here is a copy of the message that was sent to all customers on the Echo server:====Tonight at 2 AM ET [Wednesday at 2 AM ET GMT-5] we will be shutting down the 'Echo' server for approximately 15 minutes while we swap out the raid controller with a new controller.You may remember the silent corruption issues and file system checks we had to perform in the last month - these were due to faulty circuitry on the existing raid controller. We diagnosed this issue with our hardware vendor, LSI, and determined that the card was at fault.The maintenance window is from 2 AM to 3 AM ET [GMT -5], however, we expect no more than approximately 15 minutes of downtime. This process will simply entail powering down the server, pulling the existing raid controller, installing the new raid controller, and booting the system back up.Barring the system wanting to perform a file system check or any other unexpected issues the downtime should be minimal.I apologize for the short notice on this maintenance - there was some miscommunication as to when the replacement hardware would arrive and we have limited time to perform the swap under the Return Merchandise Authorization LSI has given us.If you have any questions at all about this feel free to respond to this email and we will address any and all of your questions or concerns.==== We will update this thread when the server is going offline and when it's booting back up. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2014 Author Report Share Posted February 26, 2014 The server is shutting down now for this maintenance. Quote Link to comment Share on other sites More sharing options...
iamakio Posted February 26, 2014 Report Share Posted February 26, 2014 Oh.. I hope this will fix ASAP =)). GoodLuck guys! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2014 Author Report Share Posted February 26, 2014 The server is booting up now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2014 Author Report Share Posted February 26, 2014 It did find file system errors so it is forcing a file system check. I'll keep this thread updated. http://www.screen-shot.net/2014-02-26_02-25-00.png Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2014 Author Report Share Posted February 26, 2014 It was able to recover quickly and is continuing to boot. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2014 Author Report Share Posted February 26, 2014 The server is back online, it may take a few minutes for it to stabilize but we are going to mark this maintenance as completed. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2014 Author Report Share Posted February 26, 2014 D'oh hardware team just told me there was an issue swapping the card so the original was replaced. I'll have to get with the hardware team and see what, exactly, the mix-up was as this is definitely not acceptable. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2014 Author Report Share Posted February 26, 2014 I've straightened out the issue and we are going back down to swap the card - this time the swap will happen. I'll be raising this issue with the manager of the hardware department. Quote Link to comment Share on other sites More sharing options...
ubshreenath Posted February 26, 2014 Report Share Posted February 26, 2014 Just FYI. I am waiting for the server to come back up. Don't know why but I did not receive the email about the maintenance. Checked my Spam folder too. Maybe my name's missing on the sever's mailing list. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2014 Author Report Share Posted February 26, 2014 Send me a PM with your domain. That said - the issue is that the tech doing the swap didn't know that she was supposed to swap the low-profile bracket from the old card to the new one... Seems trivial and I'm definitely going to raise it with the facility management. Anybody doing hardware swaps should have a solid idea of what they're doing and, if not, should be supervised. I'm 1,000 miles away and I knew what needed to be done and I'm not even there to see it. They should have called if they had an issue rather than putting the old card back in and booting it up. Definitely not acceptable. When it comes back up in a few minutes it should have the new card barring any other unforeseen issues. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2014 Author Report Share Posted February 26, 2014 The server is booting up now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted February 26, 2014 Author Report Share Posted February 26, 2014 The server is online - it may take a few minutes for the server to stabilize. We took a backup just before we began this maintenance and will be taking another backup very shortly [just in case]. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.