[Resolved] Echo Slowness around 1:45 to 2:00 AM EST

[Michael D.]

The backup process on the Echo server runs every morning from 1:30 AM to about 2:30 AM which does cause things to be a little slower than usual while it's running but not to the point of causing lack of responsiveness. The backup process tonight slowed things down to the point of the server becoming mostly unresponsive and as such we logged into the server within about 90 seconds to investigate. The first step we took was to disable the backup process and then rebooted the server to give it a "fresh start".

 

We discovered that the server's raid controller was running a Consistency Check on the array which would cause disk I/O performance to be decreased even without a backup process running. The Consistency Check in addition to the Backup Process together caused enough I/O delays to cause performance issues. The controller actually schedules these checks automatically by default however we've since disabled the automatic checking and have set up manual schedules for performing these checks.

 

The manual schedule is set for twice per year, or once every 6 months, as to have a minimal impact on our customers and we'll post here in the maintenance section of the forum at least 5 days in advance.

 

If you have any questions, just let us know.