Jump to content


Photo

Icarus Slowness / Issues Recently

Resolved

  • Please log in to reply
43 replies to this topic

#21 Dean

Dean

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:United Kingdom

Posted 05 August 2013 - 06:06 AM

Morning Guys,

 

Same issue again,

 

Now i am starting to get angry customers :(

 

Any Updates?


  • 0

WebHostCentral ~ Simple | Reliable | Affordable
Professional Business Hosting and Solutions.
SSL, Content Management,Social Media Management, Logos and Banners!


#22 Dehenderson

Dehenderson

    Member

  • Members
  • PipPip
  • 26 posts
  • Gender:Male
  • Location:Arlington, VA

Posted 05 August 2013 - 07:04 AM

My sites, all running WordPress, have ground to a halt. It's not possible to load the wp-admin of WordPress. I need to be making updates online and cannot because of this protracted problem with the Icarus server.

 

Update ... now, my sites are down, giving me "Error establishing a database connection."


  • 0

#23 Dean

Dean

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:United Kingdom

Posted 05 August 2013 - 07:07 AM

What happened to 24/7 support?


  • 0

WebHostCentral ~ Simple | Reliable | Affordable
Professional Business Hosting and Solutions.
SSL, Content Management,Social Media Management, Logos and Banners!


#24 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 05 August 2013 - 07:57 AM

Hello Everyone,

 

I'd like to apologize for our lack of responsivness in the forums and in tickets this morning. As you know, we are a small team, and this sometimes means we have to make a hard choice between spending our time working on an issue or responding the deluge of tickets reporting the issue. Today we opted to focus our efforts in stablizing and debugging the database issue on icarus. At this time, the MySQL server is functioning normally and your sites should be as well. We were forced to terminate the MySQL process, so you may see a moment or two of a specific database not responding as the mysqlcheck program reviews and repairs any potential corruption due to the unclean shutdown. I don't have anything new to report in terms of a cause or resolution to the problem, however we did gather extensive logs prior to shutting down the MySQL process. It will take us some time to review these, but it should give us an insight into what happened when MySQL locked up.

 

We will post another update when we have new information. In the meantime, we ask that any issues be reported via support ticket and not here on the forum. We need to keep announcement threads clean so anyone who is trying to get up to speed on the issue can easily and clearly a record of our announcements.


  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#25 100pr00f

100pr00f

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 05 August 2013 - 07:57 AM

My site is running slow but  is accessible for others to view all day so far today. (not like the last 4 days where you cant even load the site in the mornings for about 6 to 8 hours) But it is sitll slow.

 

only status that has a warning sign which has been like that for about 5 days in the cpanel is

 

Disk /dev/sda5 (/)      82 %

 

sorry to hear others still having a harder time with there site.


  • 0

#26 Dehenderson

Dehenderson

    Member

  • Members
  • PipPip
  • 26 posts
  • Gender:Male
  • Location:Arlington, VA

Posted 05 August 2013 - 08:09 AM

Sorry but I am confused ... I had submitted tickets but then was led to believe (perhaps assume) that the forum was the appropriate place for such discussion of issues. The Icarus problems are taking my time from other subjects so I will simply submit tickets, per your request.


  • 0

#27 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 05 August 2013 - 08:10 AM

My site is running slow but  is accessible for others to view all day so far today. (not like the last 4 days where you cant even load the site in the mornings for about 6 to 8 hours) But it is sitll slow.

 
The optimizations we've done so far have improved the situation, but it is still a long way off from being acceptable. That said, the timing of issues with your account doesn't seem to 100% conincide with this issue, so I would advice you open a support ticket so we can review your account to see if there is something else amiss.
 

Disk /dev/sda5 (/)      82 %

 

There is quite a bit of space left in that remaining 18%, and the server hasn't had any growth in disk usage in quite some time now.


  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#28 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 05 August 2013 - 08:16 AM

We believed the issue to be caused by R1Soft's MySQL backups and we ruled that out.  We believed the issue to be caused by some ram cache settings we adjusted a while ago - and we ruled them out.  We believed the issue was due to the R1Soft database backups - and we ruled them out.

 

I even had a tech sitting literally watching the server and it's stats all night - during the backups and for about an hour afterwards and the issue did not re-occur and we believed the issue was resolved.  He went to sleep - and I check an hour later and the issue has re-occurred.

 

I set up MySQL process logging to see what was causing this issue and, while ROOT was still locked out due to 'too may connections' this morning - our process logs show nothing.

 

There is something wrong with MySQL and we're trying to figure it out but the fact of the matter is that we didn't code/write MySQL so all we can do is troubleshoot with the tools available to us.

 

If this is causing you issues - do please open a ticket - and we'll be more than happy to move you to a server that is not experiencing his issue.

 

I realize you're frustrated - but I'm just as frustrated.  If 10 of you have 1 site down -  then I have 10 sites down.  If 100 of you have sites down, then I have 100 sites down.

 

It's not acceptable, and we're doing what we can to isolate and resolve the issue.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#29 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 05 August 2013 - 08:29 AM

Here are some graphs to give you an idea of what is happening with MySQL:

 

mysql_slowqueries-day.png

 

mysql_slowqueries-week.png

 

mysql_threads-day.png

 

mysql_threads-week.png

 

mysql_bytes-day.png

 

mysql_bytes-week.png

 

mysql_bytes-day.png

 

mysql_queries-day.png

 

mysql_queries-week.png

 

As you can see every morning MySQL is falling on it's face - we're still trying to figure out why - surely something is causing this.  It *could* be a user abusing the system but without more specific data we're not sure.  I've set up additional monitoring and tracking to help identify the issue.

 

In the meantime, if you're on the Icarus server and this has been affecting you and you want off of the server - do please simply open a ticket with the technical support department referencing this thread and asking to be transferred to a different server.  We'll be more than happy to do this for you.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#30 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 05 August 2013 - 08:41 AM

I have temporarily blocked inbound port 3306 connection requests [i.e. remote MySQL connections].  Some of our data indicates this may be a rather large brute force type of issue occurring in the mornings and jamming up our MySQL.

 

We'll know by tomorrow morning if this is the case [either the issue will re-occur, or it won't].  Since it's happening at around the same time every day I'm going to have a tech sitting and watching it just in case so we can take quick action.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#31 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 05 August 2013 - 08:42 AM

A full check/repair/optimize is running on the MySQL server now.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#32 Dean

Dean

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:United Kingdom

Posted 05 August 2013 - 08:59 AM

out of curiosity what time (your local time) is this occurring?


  • 0

WebHostCentral ~ Simple | Reliable | Affordable
Professional Business Hosting and Solutions.
SSL, Content Management,Social Media Management, Logos and Banners!


#33 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 05 August 2013 - 09:46 AM

Server time is the same as my local time - so around 4:15~4:30 AM for me.  1:15~1:30 AM for those of us in California.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#34 Dean

Dean

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:United Kingdom

Posted 05 August 2013 - 11:05 AM

So around 9:15 am for me. Its so i can send an email to clients letting them know to expect slowness etc =D

 

Thanks Michael


  • 0

WebHostCentral ~ Simple | Reliable | Affordable
Professional Business Hosting and Solutions.
SSL, Content Management,Social Media Management, Logos and Banners!


#35 Dehenderson

Dehenderson

    Member

  • Members
  • PipPip
  • 26 posts
  • Gender:Male
  • Location:Arlington, VA

Posted 05 August 2013 - 11:18 AM

Your openness, transparency, and sharing is extremely helpful. Thank you.

 

We may not be alone ... For what it's worth, I read yesterday on Twitter than many if not most of Hostgator's shared sites were down for a considerable time over the weekend.


  • 0

#36 Dean

Dean

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:United Kingdom

Posted 06 August 2013 - 05:03 AM

Looking good my end!
  • 0

WebHostCentral ~ Simple | Reliable | Affordable
Professional Business Hosting and Solutions.
SSL, Content Management,Social Media Management, Logos and Banners!


#37 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 06 August 2013 - 09:22 AM

Aye, today was good - we figured it out we believe - a combination of timed cron jobs that were fairly intensive and running at about the same time.  We've disabled some of them temporarily and will evaluate the best way to schedule them as not to have them overlap/cause issues.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#38 Dean

Dean

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:United Kingdom

Posted 06 August 2013 - 09:31 AM

Well Done Guys


  • 0

WebHostCentral ~ Simple | Reliable | Affordable
Professional Business Hosting and Solutions.
SSL, Content Management,Social Media Management, Logos and Banners!


#39 Dehenderson

Dehenderson

    Member

  • Members
  • PipPip
  • 26 posts
  • Gender:Male
  • Location:Arlington, VA

Posted 06 August 2013 - 12:27 PM

Let me echo the cheers and applause ... all seems faster than ever. Having hosted sites for 14 years, I have known of no other hosting company that responds so quickly to address and resolve issues and to provide trusted, reliable service. Believe me, MDDHosting is greatly appreciated and respected here. And, I tell everyone about your services. You are exceptional.


  • 0

#40 Dean

Dean

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:United Kingdom

Posted 07 August 2013 - 09:03 AM

I am guessing this is as good as resolved now =D Again great work Mike and Scott


  • 0

WebHostCentral ~ Simple | Reliable | Affordable
Professional Business Hosting and Solutions.
SSL, Content Management,Social Media Management, Logos and Banners!






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users