Dehenderson Posted August 5, 2013 Report Share Posted August 5, 2013 Sorry but I am confused ... I had submitted tickets but then was led to believe (perhaps assume) that the forum was the appropriate place for such discussion of issues. The Icarus problems are taking my time from other subjects so I will simply submit tickets, per your request. Quote Link to comment Share on other sites More sharing options...
Scott Posted August 5, 2013 Report Share Posted August 5, 2013 My site is running slow but is accessible for others to view all day so far today. (not like the last 4 days where you cant even load the site in the mornings for about 6 to 8 hours) But it is sitll slow. The optimizations we've done so far have improved the situation, but it is still a long way off from being acceptable. That said, the timing of issues with your account doesn't seem to 100% conincide with this issue, so I would advice you open a support ticket so we can review your account to see if there is something else amiss. Disk /dev/sda5 (/) 82 % There is quite a bit of space left in that remaining 18%, and the server hasn't had any growth in disk usage in quite some time now. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 5, 2013 Author Report Share Posted August 5, 2013 We believed the issue to be caused by R1Soft's MySQL backups and we ruled that out. We believed the issue to be caused by some ram cache settings we adjusted a while ago - and we ruled them out. We believed the issue was due to the R1Soft database backups - and we ruled them out. I even had a tech sitting literally watching the server and it's stats all night - during the backups and for about an hour afterwards and the issue did not re-occur and we believed the issue was resolved. He went to sleep - and I check an hour later and the issue has re-occurred. I set up MySQL process logging to see what was causing this issue and, while ROOT was still locked out due to 'too may connections' this morning - our process logs show nothing. There is something wrong with MySQL and we're trying to figure it out but the fact of the matter is that we didn't code/write MySQL so all we can do is troubleshoot with the tools available to us. If this is causing you issues - do please open a ticket - and we'll be more than happy to move you to a server that is not experiencing his issue. I realize you're frustrated - but I'm just as frustrated. If 10 of you have 1 site down - then I have 10 sites down. If 100 of you have sites down, then I have 100 sites down. It's not acceptable, and we're doing what we can to isolate and resolve the issue. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 5, 2013 Author Report Share Posted August 5, 2013 Here are some graphs to give you an idea of what is happening with MySQL: http://www.screen-shot.net/mysql_slowqueries-day.png http://www.screen-shot.net/mysql_slowqueries-week.png http://www.screen-shot.net/mysql_threads-day.png http://www.screen-shot.net/mysql_threads-week.png http://www.screen-shot.net/mysql_bytes-day.png http://www.screen-shot.net/mysql_bytes-week.png http://www.screen-shot.net/mysql_bytes-day.png http://www.screen-shot.net/mysql_queries-day.png http://www.screen-shot.net/mysql_queries-week.png As you can see every morning MySQL is falling on it's face - we're still trying to figure out why - surely something is causing this. It *could* be a user abusing the system but without more specific data we're not sure. I've set up additional monitoring and tracking to help identify the issue. In the meantime, if you're on the Icarus server and this has been affecting you and you want off of the server - do please simply open a ticket with the technical support department referencing this thread and asking to be transferred to a different server. We'll be more than happy to do this for you. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 5, 2013 Author Report Share Posted August 5, 2013 I have temporarily blocked inbound port 3306 connection requests [i.e. remote MySQL connections]. Some of our data indicates this may be a rather large brute force type of issue occurring in the mornings and jamming up our MySQL. We'll know by tomorrow morning if this is the case [either the issue will re-occur, or it won't]. Since it's happening at around the same time every day I'm going to have a tech sitting and watching it just in case so we can take quick action. Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 5, 2013 Author Report Share Posted August 5, 2013 A full check/repair/optimize is running on the MySQL server now. Quote Link to comment Share on other sites More sharing options...
Dean Posted August 5, 2013 Report Share Posted August 5, 2013 out of curiosity what time (your local time) is this occurring? Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 5, 2013 Author Report Share Posted August 5, 2013 Server time is the same as my local time - so around 4:15~4:30 AM for me. 1:15~1:30 AM for those of us in California. Quote Link to comment Share on other sites More sharing options...
Dean Posted August 5, 2013 Report Share Posted August 5, 2013 So around 9:15 am for me. Its so i can send an email to clients letting them know to expect slowness etc =D Thanks Michael Quote Link to comment Share on other sites More sharing options...
Dehenderson Posted August 5, 2013 Report Share Posted August 5, 2013 Your openness, transparency, and sharing is extremely helpful. Thank you. We may not be alone ... For what it's worth, I read yesterday on Twitter than many if not most of Hostgator's shared sites were down for a considerable time over the weekend. Quote Link to comment Share on other sites More sharing options...
Dean Posted August 6, 2013 Report Share Posted August 6, 2013 Looking good my end! Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 6, 2013 Author Report Share Posted August 6, 2013 Aye, today was good - we figured it out we believe - a combination of timed cron jobs that were fairly intensive and running at about the same time. We've disabled some of them temporarily and will evaluate the best way to schedule them as not to have them overlap/cause issues. Quote Link to comment Share on other sites More sharing options...
Dean Posted August 6, 2013 Report Share Posted August 6, 2013 Well Done Guys Quote Link to comment Share on other sites More sharing options...
Dehenderson Posted August 6, 2013 Report Share Posted August 6, 2013 Let me echo the cheers and applause ... all seems faster than ever. Having hosted sites for 14 years, I have known of no other hosting company that responds so quickly to address and resolve issues and to provide trusted, reliable service. Believe me, MDDHosting is greatly appreciated and respected here. And, I tell everyone about your services. You are exceptional. Quote Link to comment Share on other sites More sharing options...
Dean Posted August 7, 2013 Report Share Posted August 7, 2013 I am guessing this is as good as resolved now =D Again great work Mike and Scott Quote Link to comment Share on other sites More sharing options...
Scott Posted August 7, 2013 Report Share Posted August 7, 2013 I am guessing this is as good as resolved now =D Pretty much. I'm trying to put together some additional info about the situation to post here as my free time allows, but I'm once again wearing a brace on my left hand for carpal tunnel, and typing is now a somewhat slow/laborious/uncomfortable process. Using a voice-to-text program is about the same, especially with technical terms. (Cache vs Cash, etc) Quote Link to comment Share on other sites More sharing options...
Dean Posted August 7, 2013 Report Share Posted August 7, 2013 Take your time Scott! Aslong as the problem is solved Info can come later. P.S (stop sleeping on your side =D) Quote Link to comment Share on other sites More sharing options...
Dehenderson Posted August 16, 2013 Report Share Posted August 16, 2013 Hi all, Just curious if Icarus is back to normal and the issues resolved. All of my sites on Icarus seem to be operating perfectly. I notice that in the Pingdom report provided on this forum, Icarus is the only server that's grayed out with a "?" in the center of the gray. Thanks,David Quote Link to comment Share on other sites More sharing options...
Michael D. Posted August 17, 2013 Author Report Share Posted August 17, 2013 Icarus has been back to normal but it looks like I forgot to unpause the external monitoring. Thankfully we have internal monitoring that we rely on - the external is just for customers that do not have their own monitoring set up . I unpaused it now. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.