Michael D. Posted February 18, 2011 Report Share Posted February 18, 2011 The echo server's raid controller kicked off an automatic Battery Back Up learn cycle which disables write caching and at this point is creating disk I/O issues slowing things down. We've gotten with the controller vendor (LSI) and they've unfortunately let us know that there is no way to stop a learn cycle once it's started and that doing so is a very bad idea. What the learn cycle is designed to do is to drain and refill the battery backup of the raid controller so that in the event that it's actually needed, such as an unexpected power failure, that the battery will have enough power to write any cached data to the drives so that the file system does not get corrupted resulting in downtime and data loss beyond the power outage itself. Without the learn cycle, over time, the battery will lose capacity without the controller being aware of the degraded capacity which could result in extreme data corruption in the event of an unexpected power outage. We have scheduled all of our servers to run this cycle at night when the disk usage is the lowest to minimize any impact however the Echo server kicked off this learning process mid-afternoon today. The process itself shouldn't last more than 4 hours as the battery drains and refills and we've disabled the system from doing this automatically in the future to prevent it happening during the day or other peak times. We apologize for any issues this may have caused you and we assure you that we've taken steps to make sure that this specific issue does not occur again. Thank you, Link to comment Share on other sites More sharing options...
Michael D. Posted February 18, 2011 Author Report Share Posted February 18, 2011 The learn cycle is still running however the raid controller has "caught up" to the demand even without the cache and as of now everything is just about at normal speed (maybe a tid bit slower than usual). Link to comment Share on other sites More sharing options...
Michael D. Posted February 18, 2011 Author Report Share Posted February 18, 2011 We've done some investigation on our own and have made some changes to prevent this from happening again in the future across all servers.We've disabled the auto-learn cycle on the controller.This means that the system will no longer kick into a learn cycle on it's own.We've set up scheduled learn cycles once every 3 months for off-peak times.This means that the learn cycle will still be run once every 3 months as it's an important task, but we've manually scheduled these learn cycles for off-peak times as not to impact server performance. Link to comment Share on other sites More sharing options...
Michael D. Posted February 18, 2011 Author Report Share Posted February 18, 2011 The learn cycle has completed and the server is now back to normal. Link to comment Share on other sites More sharing options...
iansltx Posted February 28, 2011 Report Share Posted February 28, 2011 Echo appears to be really slow right now again. Load is 8+...what's going on?!? EDIT: Slow loads are gone now. Link to comment Share on other sites More sharing options...
Michael D. Posted February 28, 2011 Author Report Share Posted February 28, 2011 Echo appears to be really slow right now again. Load is 8+...what's going on?!?Echo is fine, and a load of up to 16 is fine (as it's a 16 core server). I'd suggest that you open a ticket. Link to comment Share on other sites More sharing options...
Adam Posted March 6, 2011 Report Share Posted March 6, 2011 It does seem like Echo is slowing down. Yesterday and today, the server slowed to a crawl for a few minutes and the server status page said that there was 0% cpu usage. I know what happened yesterday, but what about today? Link to comment Share on other sites More sharing options...
Michael D. Posted March 6, 2011 Author Report Share Posted March 6, 2011 It does seem like Echo is slowing down. Yesterday and today, the server slowed to a crawl for a few minutes and the server status page said that there was 0% cpu usage. I know what happened yesterday, but what about today?The server backup process runs from 12:30 AM to around 1:30 AM GMT-5 on the server every night - during this period things will be a *little* more sluggish than normal but being that it's during an off-peak time and it's the backups it's simply a part of running a web server. Alternatively we could not back the server up at all and should a major hardware failure happen, or worse, and you don't have your own backups all of your data would be gone. Having a slight increase in latency for 1 hour during an off-peak time so that we can keep a complete server replica in case of a major issue is a small trade-off. Ultimately if you cannot ever have any sluggishness due to backups, or any other reason, I would suggest a dedicated server where you can be in complete control of what processes are or are not running and when they run. I can't say I know of any accounts on our shared platform that need a dedicated server so that would be your own personal decision. This thread was opened for a very specific issue and that issue was resolved quite a while ago. Since people seem to keep responding to this thread for unrelated issues I'm going to go ahead and close this thread. If you feel the server you are on is preforming slowly or is having issues by all means, do open as upport ticket. Thanks! Link to comment Share on other sites More sharing options...
Recommended Posts