Jump to content


Photo

S1 / R1 Servers - Network Device Updates - ~9 PM ET Jan 12, 2016.

Scheduled

  • Please log in to reply
42 replies to this topic

#21 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 18 January 2016 - 12:53 PM

The server has initiated a dump and a restart.  we are investigating the cause.


  • 0

#22 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 18 January 2016 - 01:03 PM

We have found the cause of the restart loop.  Our backup software was causing a kernel panic and causing the server to initiate a reboot.  it has been disabled and the server is currently online.


  • 0

#23 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 18 January 2016 - 01:06 PM

I am currently looking into litespeed issues that are causing pages not to load.


  • 0

#24 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 18 January 2016 - 01:08 PM

The server has kernel panic'd again.. Sadly, r1soft was not the core cause.  we are continuing to search for the reason behind this fault.


  • 0

#25 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 January 2016 - 01:08 PM

The system is out of the FSCK loop, however, it is producing kernel panics on boot now.  We disabled one kernel module that we believed was responsible and it was online longer this time around.  We're still working on this.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#26 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 January 2016 - 01:18 PM

The server is back online, however, I am not sure I can call it stable.

 

I'm going to get back to work on investigating the root cause of the instability so we can resolve this once and for all.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#27 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 18 January 2016 - 01:20 PM

I am working on fixing sites with missing content while continuing to search for the source of the restarts.  at this time the restarts have slowed down significantly. I do have multiple debug sessions open watching different aspects of the server.


  • 1

#28 mdd_shared_user

mdd_shared_user

    Newbie

  • Members
  • Pip
  • 8 posts

Posted 18 January 2016 - 01:27 PM

Thanks for the updates and fire fighting.  Just FYI, here's what I'm seeing when trying to load my website using Firefox:

 

Content Encoding Error

The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression.

    Please contact the website owners to inform them of this problem.


  • 0

#29 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 18 January 2016 - 01:39 PM

Can you please open a ticket for that specific issue?  That is not a issue that is being seen across other sites at this time.


  • 0

#30 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 18 January 2016 - 02:14 PM

The server is still online and we are spining up secondary servers to recreate the problem with.  
Also, if you continue to have websites with content issues please open or reply to your existing tickets.  We have addressed the filesystem glitch that was causing some files not to load as well as the content encoding errors.


  • 0

#31 Plippers

Plippers

    Newbie

  • Members
  • Pip
  • 16 posts
  • Gender:Male

Posted 18 January 2016 - 08:38 PM

Hey Guys,

 

When you first announced upgrades the other month, I think you mentioned more info to follow.

 

I'm just wondering if this thread is all the info on the new setup? I'm excited to see what you're rolling out and am wondering if there's an email blast to customers to follow?

 

Cheers!


  • 0

#32 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 January 2016 - 09:11 PM

We've been working extremely hard all day on resolving this issue.  I am going to be bringing down the S1 server momentarily to bring it online on a different piece of hardware.

 

The total downtime should not exceed 10 minutes although I expect it to be more around 3 minutes.  I'll update this once it's going down and once it's back up.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#33 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 January 2016 - 09:12 PM

Bringing S1 down now.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#34 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 18 January 2016 - 09:25 PM

S1 is back online.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#35 Leah2

Leah2

    Member

  • Members
  • PipPip
  • 29 posts
  • Gender:Not Telling

Posted 18 January 2016 - 09:42 PM

Hi - as always MDD's communication is incredible!

 

Question - I've been getting pingdom up & down reports all day that coincide with the S1 server for the Gemini server. Are the 2 somehow tied together? I'm not getting any downtime reports from the other sites on other servers @ MDD I monitor.

 

It's been confusing because the site has been offline according to the uptime robot & my trying to access the website via browser & MDD's server status page. Yet the public report on Gemini shows 100% uptime.

 

Thanks!


  • 0

Electronic Logic Concepts

 

“What is Your Digital Strategy? Websites Built With SEO First Practices”

 

www.ELC-SEO.com


#36 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 19 January 2016 - 02:55 AM

Gemini accounts were moved to S1 a little while ago - you would have received an email about this and can see a copy of it here:

https://www.mddhosti...p?action=emails

If you can't find it do please open a ticket.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#37 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 19 January 2016 - 03:01 AM

I do apologize to everybody that experienced any issues or downtime as a result of these issues.

 

We are making a huge move to vastly improved hardware that is already as of now allowing us to allow far more redundancy and speed than we could before.  If the hardware that the S1 server is running on were to completely fail catastrophically the server would come back online on another piece of hardware within a few minutes - far faster than we could have ever handled hardware failure before today.

 

Unfortunately such a big change has clearly caused us some growing pains - all of which I do believe should be resolved at this point.  Some of it required working with our software vendors to resolve issues we're experiencing and some of it involved network consulting.

 

While I cannot promise you won't have issues / there won't be problems - what I can promise you is that if they do happen we will be working on them most likely before you have a chance to notice yourself.  We are monitoring our new servers by the second where the old hardware was monitored by the minute.  We are still human and may not respond within a second but it does greatly decrease our response time to incidents that may occur.

 

Thank you for your patience and understanding in this process and again I'm sorry for any trouble you may have experienced as a result.  We don't like outages or downtime anymore than you do and these changes ultimately should result in the absolutely maximum uptime we could provide for a company of our size.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#38 skunkbad

skunkbad

    Member

  • Members
  • PipPip
  • 26 posts

Posted 19 January 2016 - 11:00 AM

What's the schedule for Demeter?


  • 0

#39 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 19 January 2016 - 11:07 AM

What's the schedule for Demeter?

You will receive a notice 72 hours prior to us getting started with Demeter.  At this point the only schedule for demeter is that it's not yet scheduled.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#40 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 19 January 2016 - 01:15 PM

The S1 server is still experiencing short outages - but not due to the same/original issue.  I believe this may simply be a CentOS7/CloudLinux7 issue but I am actively working on investigating this.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users