Jump to content


Photo

Icarus Slowness / Issues Recently

Resolved

  • Please log in to reply
43 replies to this topic

#1 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 August 2013 - 08:09 AM

Hello.

 

As a hosting provider that relies upon quality of service and word of mouth for growth it's obviously not in our best interests to have a server that's performing poorly or having issues.  That said - we have a solid track record of keeping things fast, stable, and resolving issues quickly.

 

The Icarus server, unfortunately, has been facing some issues as a result of the R1Soft backup system.  This system, when operating normally, only copies the data changed from day to day - so while the server may have 500 GB of data stored, will only transmit the 3 to 5GB of data that actually *changed* from day to day.

 

When there is an issue such as a power failure, a server reboot, or anything else that causes the backup agent on the server to get restarted - it has no way of knowing what has changed and what has not.  In this sort of situation the backup server treats the server almost like it's new - it reads each and every block to determine what has or has not changed - it's a very intensive process that takes more time than a usual daily backup.

 

Additionally if something goes wrong with the backup process - it can get 'stuck' and cause issues.  For three days in a row we were diagnosing the backup system as it was indeed getting suck and causing issues.  We believe we resolved the core issue causing this yesterday and had everything ready to go, however, our facility had a minor power outage to a part of the data center floor affecting both the backup server and the Icarus server.  Due to this unexpected outage the backup server had to do another fresh backup last night.

 

During this backup - the server was slower than usual.  R1Soft is supposed to be able to be run on a server without adversely affecting it's performance but it seems this is not the case with the Icarus server.  At the end of the day our newer hardware is more robust and handles load/backups better than our older hardware - and we're in the process of upgrading our hardware but we're not done with that yet.

 

If this issue continues to persist with the Icarus server we'll be forced to migrate everybody off of it - however - we would like to do this as a part of our normal upgrade schedule.  We will, however, perform migrations sooner rather than later if the backup system continues to cause issues for the Icarus server.

 

We do deeply apologize for any issues the disruptions may have caused you.  We do perform the backups during the least busy time for the server, however, that doesn't mean that the server is idle so it does still impact some customers.  When faced with the option of having a couple of hours of slowness versus having no backups at all we choose to make sure we, at the least, have a safe backup copy of your data.

 

Barring any unexpected issues - the server should perform a normal incremental backup tomorrow night and it should not cause issues.

 

If you have any questions, by all means, feel free to ask.


Edited by Scott S, 07 August 2013 - 09:09 AM.
Marked resolved.

  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#2 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 August 2013 - 09:25 AM

Only MySQL at this time appears to be running slow - we're investigating.

 

I have issued a restart for MySQL just now which normally takes 5 to 10 seconds but is taking longer than expected.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#3 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 August 2013 - 09:29 AM

The restart is still in progress - there is definitely something wrong with MySQL and we're going to continue to investigate.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#4 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 August 2013 - 09:31 AM

MySQL is back online.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#5 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 August 2013 - 09:36 AM

After the restart MySQL appears to be performing normally.  I suspect some optimization needs to be performed - normally I would handle such optimizations but I will likely bring in a trusted administrator that is much more experienced than myself to give it a look due to the issues we've faced over the last few days.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#6 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 03 August 2013 - 10:06 AM

I have made some optimizations on the server that should help prevent this issue from reoccurring.  We're going to monitor the server, however, feel free to report if you feel anything is behaving abnormally.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#7 Dean

Dean

    Member

  • Members
  • PipPip
  • 25 posts
  • Gender:Male
  • Location:United Kingdom

Posted 04 August 2013 - 06:15 AM

Hello Michael.

 

MySQL is experiencing some issue on my websites. Thanks


  • 0

WebHostCentral ~ Simple | Reliable | Affordable
Professional Business Hosting and Solutions.
SSL, Content Management,Social Media Management, Logos and Banners!


#8 Battra

Battra

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 04 August 2013 - 07:51 AM

Right now I'm having problem accessing my sites. The server itself is up but it's either extremely slow or not loading at all. 


  • 0

#9 100pr00f

100pr00f

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 04 August 2013 - 08:25 AM

same thing here. My site has been almost down for about 3 days due to its slowness and my viewers and readers cant access the site at all.


  • 0

#10 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 04 August 2013 - 09:00 AM

Hey guys,

 

Sorry I didn't catch this earlier - while the server performed better this morning it still was not up-to-par.  I've adjusted the settings more and the issue shouldn't reoccur tomorrow.  I did set up some special monitoring for this to wake me up if the issue reoccurred but it seems the site I chose to monitor wasn't intensive enough to be affected.

 

The issue is only occurring when the R1Soft Backups [daily backups] are running. Recently some changes to R1Soft have caused it to be a little more intensive than usual resulting in slowness during the backup process. We do schedule the backups overnight to have the least impact but even during off-peak times we need our servers to perform well.

 

The changes I've made will permit more data that needs written to the disks to be cached into RAM - this helps as during the backups the reads for the backups take priority and writes can be severely delayed resulting in slowness/downtime.  I've set it substantially higher [upwards of 20% of system ram or 4.8 GB] which should be plenty to keep the server stable during backups.

 

We do plan on replacing this hardware with much more robust hardware as soon as possible - likely within then next 6 months. The newer hardware can handle R1Soft backups mid-day even when the server is busy without anybody noticing. We're just a smaller company so we have to work towards that higher-end and every expensive hardware :).


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#11 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 04 August 2013 - 09:11 AM

It appears R1Soft may be causing issues with MySQL.

 

For example:

root@icarus [~]# mysqladmin status
mysqladmin: connect to server at 'localhost' failed
error: 'User root already has more than 'max_user_connections' active connections'

 

There is nothing going on now that would cause root to use more than a couple of connections.  I'm restarting MySQL now.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#12 MikeDVB

MikeDVB

    Forum Administrator

  • Staff Administrator
  • PipPipPipPipPip
  • 2,900 posts
  • Gender:Male
  • Location:Central Indiana, USA

Posted 04 August 2013 - 09:17 AM

MySQL is back - if this issue occurs tomorrow [MySQL messing up] we'll have to disable the database-specific backups on this server.  I'll set up an alarm tomorrow for me to check the server *as* the backup runs so I can see what is happening at the time of the failure.


  • 0
Michael Denney - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#13 100pr00f

100pr00f

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 04 August 2013 - 09:31 AM

thanks


  • 0

#14 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 04 August 2013 - 12:52 PM

same thing here. My site has been almost down for about 3 days due to its slowness and my viewers and readers cant access the site at all.

 

If your site has been down all this time, the cause isn't the same as sites should be back to normal as soon as the backup stops or MySQL is restarted. Please open a support ticket so we can investigate your site specifically.


  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#15 kothet

kothet

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 04 August 2013 - 10:59 PM

Here is what happened to me.

 

Yesterday was 5th day that I had this problem. Started since 31st July. I opened the ticket in day 1, 2, 3 and 4 and I don't bother opening ticket yesterday anymore.

 

Everyday, at the same time, the site went down or being very very slow. Support didn't even notice they had problem until I opened ticket in Day 1 and 2. In one ticket, they even said site is fine and probably my ISP. When I said my clients had same problem from other countries, then they started to send ticket to level 2 support, who took about 3 hours in day 2 and even longer in day 4.

 

Even my free monitor system alerted me about this problem right away and you guys didn't seems to notice until I opened the ticket. You said you will try to migrate the sites to another server when it happened again. It did happened again yesterday and you still have no solution.

 

This is too much for mddhosting for a company who guarantee 99/9% uptime. Although you don't even mention about compensating your valued customers, I do have to compensate my value clients who I care about.

 

5 straight days with same problem and hours to fix the same issue? I am moving all my accounts to another reliable host. My mistake about MDD hosting.

 

BTW, I can be patience up to 3 days. Not 5 days. If today happened, it will be 6th. 


  • 0

#16 Scott

Scott

    MDDHosting Staff

  • Staff Administrator
  • PipPipPipPip
  • 421 posts
  • Gender:Male

Posted 05 August 2013 - 01:55 AM

Backups have started on the icarus server as of a few minutes ago. Everything appears stable, but we will continue to monitor the server until backups have completed.


  • 0
Scott S - MDDHosting LLC - Providing Hosting since 2007
Scalable shared hosting plans in the cloud! Check them out!
Highly Available Cloud Shared, Reseller, and VPS
http://www.mddhosting.com/

#17 kothet

kothet

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 05 August 2013 - 04:22 AM

its happening again i think. Loading slow and my monitors are sending me alerts...


  • 0

#18 luan152

luan152

    Newbie

  • Members
  • Pip
  • 5 posts

Posted 05 August 2013 - 04:38 AM

Yep, my websites are slow, too.


  • 0

#19 Battra

Battra

    Newbie

  • Members
  • Pip
  • 4 posts

Posted 05 August 2013 - 05:32 AM

Any update on this issue? My sites are not responding again ...


  • 0

#20 kothet

kothet

    Newbie

  • Members
  • Pip
  • 3 posts

Posted 05 August 2013 - 05:52 AM

6 days straight. Almost 2 hours since the problem started today and no solution yet. I am moving my client sites to another host since this morning and I think i made the right decision.


  • 0





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users