Jump to content


Photo

Echo Server - Scheduled Maintenance - 2 AM ET on 02/26/2014

Completed

  • Please log in to reply
12 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 01:08 AM

Here is a copy of the message that was sent to all customers on the Echo server:

====

Tonight at 2 AM ET [Wednesday at 2 AM ET GMT-5] we will be shutting down the 'Echo' server for approximately 15 minutes while we swap out the raid controller with a new controller.

You may remember the silent corruption issues and file system checks we had to perform in the last month - these were due to faulty circuitry on the existing raid controller.  We diagnosed this issue with our hardware vendor, LSI, and determined that the card was at fault.

The maintenance window is from 2 AM to 3 AM ET [GMT -5], however, we expect no more than approximately 15 minutes of downtime.  This process will simply entail powering down the server, pulling the existing raid controller, installing the new raid controller, and booting the system back up.

Barring the system wanting to perform a file system check or any other unexpected issues the downtime should be minimal.

I apologize for the short notice on this maintenance - there was some miscommunication as to when the replacement hardware would arrive and we have limited time to perform the swap under the Return Merchandise Authorization LSI has given us.

If you have any questions at all about this feel free to respond to this email and we will address any and all of your questions or concerns.

====

 

We will update this thread when the server is going offline and when it's booting back up.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#2 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 02:03 AM

The server is shutting down now for this maintenance.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#3 iamakio

iamakio

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 26 February 2014 - 02:17 AM

Oh.. I hope this will fix ASAP =)). GoodLuck guys!


  • 0

#4 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 02:24 AM

The server is booting up now.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#5 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 02:24 AM

It did find file system errors so it is forcing a file system check.

 

I'll keep this thread updated.

 

2014-02-26_02-25-00.png


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#6 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 02:26 AM

It was able to recover quickly and is continuing to boot.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#7 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 02:28 AM

The server is back online, it may take a few minutes for it to stabilize but we are going to mark this maintenance as completed.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#8 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 02:29 AM

D'oh hardware team just told me there was an issue swapping the card so the original was replaced.

 

I'll have to get with the hardware team and see what, exactly, the mix-up was as this is definitely not acceptable.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#9 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 02:32 AM

I've straightened out the issue and we are going back down to swap the card - this time the swap will happen.  I'll be raising this issue with the manager of the hardware department.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#10 ubshreenath

ubshreenath

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 26 February 2014 - 02:38 AM

Just FYI. I am waiting for the server to come back up. Don't know why but I did not receive the email about the maintenance. Checked my Spam folder too. Maybe my name's missing on the sever's mailing list. 


  • 0

#11 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 02:39 AM

Send me a PM with your domain.

 

That said - the issue is that the tech doing the swap didn't know that she was supposed to swap the low-profile bracket from the old card to the new one... Seems trivial and I'm definitely going to raise it with the facility management.  Anybody doing hardware swaps should have a solid idea of what they're doing and, if not, should be supervised.

 

I'm 1,000 miles away and I knew what needed to be done and I'm not even there to see it.  They should have called if they had an issue rather than putting the old card back in and booting it up.  Definitely not acceptable.

 

When it comes back up in a few minutes it should have the new card barring any other unforeseen issues.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#12 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 02:49 AM

The server is booting up now.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#13 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 26 February 2014 - 02:53 AM

The server is online - it may take a few minutes for the server to stabilize.

 

We took a backup just before we began this maintenance and will be taking another backup very shortly [just in case].


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users