Jump to content


Photo

S1 Server - Random Lag/Inaccessibility + Short Outage

Current

  • This topic is locked This topic is locked
36 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 March 2016 - 12:08 PM

We have, unfortunately, been experiencing issues with the S1 server over the last week. We believe the issue to be in the LiteSpeed Web Server Software on S1.

We have configured the server to switch to Apache+LSAPI [keeping many of the benefits of LiteSpeed, while using Apache at the core]. This should have been seamless but I personally failed to communicate this to our team effectively resulting in them working to restore LSWS when I switched it to Apache.

 

Ultimately the miscommunication on my part is the reason for the outage.

 

We are testing/monitoring the server's performance with Apache 2.4 + MPM Event.  One of the primary reasons we switched away from Apache many years ago was due to how resource intensive and bloated Apache was to accomplish the same goals.  At the time we were limited to systems with 2 to 8 GB of ram so being conservative of resources was a huge benefit and being faster was great as well.

 

At this point we're running machines with 100GB of ram or more - often 20 to 50GB of ram just not doing anything at all.  As such we don't mind so much the extra resource usage of Apache if it performs better/more reliable than LiteSpeed.

 

Many of the issues we've faced over the last 6 months have been LiteSpeed related - be it compatible with CentOS7 or simply bugs we've found and had to work to have resolved.

 

If you have any issues with any of your sites please let us know.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#2 LasVegasRealEstate

LasVegasRealEstate

    Newbie

  • Members
  • Pip
  • 1 posts

Posted 21 March 2016 - 12:21 PM

I'm not sure if I am hosted on this server but my guess is yes. It appears wildcard redirects in cpanel break with this change. Two of my many domains have wildcard redirects which were working fine until now. They end up in an endless loop with the site not displaying. I removed the wildcards and it works again. Just an FYI. Maybe it isn't related.


  • 0

#3 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 March 2016 - 01:26 PM

I'm not sure if I am hosted on this server but my guess is yes. It appears wildcard redirects in cpanel break with this change. Two of my many domains have wildcard redirects which were working fine until now. They end up in an endless loop with the site not displaying. I removed the wildcards and it works again. Just an FYI. Maybe it isn't related.

What that more than likely means is you have a syntax error in your mod_rewrite or something else in your .htaccess that LiteSpeed was ignoring that Apache won't.  One thing LiteSpeed is very good at is glossing over issues/errors when it comes to .htaccess files while Apache is much more strict.

 

Do open a support ticket if you don't have one already and we'll take a look for you.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#4 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 March 2016 - 02:05 PM

We identified an issue affecting SSL on the S1 server and it has been resolved.

 

We configure our servers so that if you try to connect to SSL on a domain that doesn't have SSL installed - you see a default page / error page instead of somebody else's site that *does* have SSL and is on the same IP.

Unfortunately this default error page was compatible with LiteSpeed but not Apache.  As soon as we were able to identify this as the issue we reverted it and SSL is working as expected.

I am sorry for any trouble this has caused you.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#5 SarisIsop

SarisIsop

    Advancing Member

  • Members
  • PipPipPip
  • 156 posts
  • Gender:Not Telling

Posted 21 March 2016 - 02:24 PM

I still have one site (mentioned in ticket) not redirecting properly.


  • 0

#6 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 21 March 2016 - 04:55 PM

I apologize for the short outage.  Upon reactivating modsecurity apache refused to start.  I have removed modsecurity and will be working to determine the cause of the issues.


  • 0

#7 ericr

ericr

    Staff

  • Staff Administrator
  • PipPipPip
  • 224 posts
  • Gender:Male

Posted 21 March 2016 - 04:56 PM

I'm not sure if I am hosted on this server but my guess is yes. It appears wildcard redirects in cpanel break with this change. Two of my many domains have wildcard redirects which were working fine until now. They end up in an endless loop with the site not displaying. I removed the wildcards and it works again. Just an FYI. Maybe it isn't related.

 

As for your issues, can you please open a ticket so that we can investigate.


  • 0

#8 haus

haus

    Newbie

  • Members
  • Pip
  • 2 posts

Posted 21 March 2016 - 07:27 PM

I'm using AddHandler application/x-httpd-php .php .html .htm in my .htaccess file and none of my sites on S1 are serving up PHP, they're just passing the code to the browser in  plaintext. Did you make a change to the config? I don't recall getting a note about it. This is pretty serious, half my sites aren't working.


  • 0

#9 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 21 March 2016 - 07:38 PM

I'm using AddHandler application/x-httpd-php .php .html .htm in my .htaccess file and none of my sites on S1 are serving up PHP, they're just passing the code to the browser in  plaintext. Did you make a change to the config? I don't recall getting a note about it. This is pretty serious, half my sites aren't working.

 

Yes, the changes made today were quiet serious, so it's likely that your issue is related. If you haven't already, please open a ticket so we can review your specific configuration and better advise/help you.


  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#10 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 21 March 2016 - 09:10 PM

The issue that just occurred now affected all servers - so I will be opening a new thread for it.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#11 Vilandra

Vilandra

    Newbie

  • Members
  • Pip
  • 23 posts
  • Gender:Female
  • Location:Pittsburgh, PA
  • Interests:Chelsea FC!

Posted 21 March 2016 - 09:16 PM

I couldn't have been happier back when I was on Hermes...barely ever had a problem if ever...but ever since the migration it's just not been the same.


  • 0

#12 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 21 March 2016 - 09:31 PM

I couldn't have been happier back when I was on Hermes...barely ever had a problem if ever...but ever since the migration it's just not been the same.

 

As the saying goes, "when it rains, it pours," and it seems El Nino has reached MDDHosting... so to speak. We recognize all of the issues have taken us below our standards and promise we are working to get everyone back to normal as soon as possible. I do want to apologize for the problems and thank you for your patience while we get everything fixed up.

 

Please don't hesitate to open a support ticket if you find your sites aren't working as expected.


  • 1
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#13 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 March 2016 - 12:34 AM

We did resolve a rather large issue that was resulting in MariaDB [MySQL] not performing up to our expectations.  Once we were able to sort this issue the server has been much faster and more reliable.

 

Every customer that opened a ticket about slowness/lag that I could find a ticket for has reported so far that things are fast again.

 

That said if you are having issues and you are on the S1 server do please open a ticket so we can investigate.

 

Sorry for any trouble this may have cause you.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#14 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 March 2016 - 08:29 AM

S1 is performing better than it has but it is still not up to our standards.

 

I'm bringing in a trusted third party server administrator to give this a fresh look / a new set of eyes.  I've worked with this administrator in the past when I was absolutely completely stuck and he is a wizard.

 

I'll update this thread when I have more information.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#15 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 March 2016 - 09:00 PM

Just sent this email:

----

I first and foremost would like to apologize for the issues you have faced on the S1 server over the last couple of weeks.

 

We are going to be rebooting the server within the next 30 minutes to change the way CPU is assigned to the server.

I am hesitant to say that this will resolve the issues the server has been facing but I am optimistic. We will be continuing to monitor the server closely and working to ensure that the server performs to your expectations.

Again I am sorry for any trouble you've faced thus far and I am sorry that we do have to initiate this reboot. So long as we don't run into any unexpected issues the total downtime should be less than 2 minutes.

If you have any questions or concerns please do not hesitate to reach out.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#16 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 March 2016 - 09:02 PM

I should know with my luck here lately that things wouldn't go as smoothly as planned.

 

The file system has errors and a file system check is being forced by the OS.  This is ultimately a good thing as it will resolve file system inconsistencies and prevent data loss and corruption in the long term.

 

I will keep this thread updated with status updates.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#17 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 March 2016 - 09:12 PM

We rebooted into a maintenance mode to run the FSCK manually - this allows us to provide a progress status.

 

It is 10.3% completed according to the status.  It lists no ETA but I will keep this thread updated.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#18 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 March 2016 - 09:16 PM

19.5% Completed


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#19 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 March 2016 - 09:19 PM

File System Check 26% Completed


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#20 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 23 March 2016 - 09:21 PM

30% Completed


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users