Michael D. Posted March 24, 2016 Author Report Share Posted March 24, 2016 It moved on to checking the directory structure and it appears this portion is going to take longer. It is currently at 71.8% Link to comment Share on other sites More sharing options...
Michael D. Posted March 24, 2016 Author Report Share Posted March 24, 2016 The file system check has reached 73% Completed. Link to comment Share on other sites More sharing options...
Michael D. Posted March 24, 2016 Author Report Share Posted March 24, 2016 The file system check has reached 75% Link to comment Share on other sites More sharing options...
Michael D. Posted March 24, 2016 Author Report Share Posted March 24, 2016 The file system check has reached 80%. Link to comment Share on other sites More sharing options...
Michael D. Posted March 24, 2016 Author Report Share Posted March 24, 2016 The File System Check has reached 87% completed. Link to comment Share on other sites More sharing options...
Michael D. Posted March 24, 2016 Author Report Share Posted March 24, 2016 The file system check has completed and the server is booting up. Link to comment Share on other sites More sharing options...
Scott Posted March 24, 2016 Report Share Posted March 24, 2016 S1 is now back online and handling requests "normally." The reboot was initially meant to correct a configuration issue, which we are now monitoring to see if the previous stability problems are resolved. I should also note that S1 may be slower for the next 10 or 20 minutes, as most servers of this size are when they come back online. 1 Link to comment Share on other sites More sharing options...
Michael D. Posted March 24, 2016 Author Report Share Posted March 24, 2016 So far this morning the performance of the S1 server has been very much improved. I'm hesitant to call it 'fixed' as I want to allow more time to pass before we come to that determination. I'm highly optimistic, however, that the issues are resolved. Link to comment Share on other sites More sharing options...
Michael D. Posted March 24, 2016 Author Report Share Posted March 24, 2016 Other than a MySQL restart issued by me at ~1:15 PM today the server has been rock solid and performing well. We do not anticipate any further issues with the S1 server at this time. Link to comment Share on other sites More sharing options...
Michael D. Posted March 25, 2016 Author Report Share Posted March 25, 2016 The R1Soft System Kernel caused a panic on S1. We're rebooting and working to restore services as soon as possible. As a result of this issue we're going to be disabling R1Soft on this server for at least a couple of days while we investigate/evaluate our options. Link to comment Share on other sites More sharing options...
Scott Posted March 25, 2016 Report Share Posted March 25, 2016 The reboot is complete and sites are responding once again. We are still investigating the source of the kernel panic with R1soft and have no further details regarding that at this time. Link to comment Share on other sites More sharing options...
Michael D. Posted March 25, 2016 Author Report Share Posted March 25, 2016 Interestingly enough R1Soft managed to load up the storage subsystem enough that we identified a previously unknown performance bottleneck. We made an adjustment during the outage and average storage latency is down to 1ms from 10ms after this reboot. We made the same adjustment live to the rest of the fleet. Hopefully this will be the good that comes from the bad at this point. Link to comment Share on other sites More sharing options...
Recommended Posts