S3 - Recent Intermittent Brief Outages

[Michael D.]

We identified an issue where 'doveadm', the tool used as a part of dovecot the IMAP mail server, in a specific situation is able to go into an endless loop using more and more ram. This single process in a single account was up to 20GB of RAM usage alone - which is enough to put pressure on the system to the point things slow down and become very slow to respond.

 

http://www.screen-shot.net/2018-10-20_04-18-35.png

 

We have reached out to the account owner to inform them of this - not that it is specifically their fault - but so that they are aware of what is going on and the changes we've made to their account.

 

We're also reporting this issue to cPanel as they are providing us with Dovecot and reporting this to them is, I believe, the best course of action to get this behavior changed/fixed. While it will likely need to be Dovecot developers themselves that fix this - cPanel would have a working relationship with them and can likely get this fixed faster than we can.

 

Additionally we're reaching out to CloudLinux about this as it should not be possible for a process owned by a user to consume this much system memory.

 

Beyond that we've also made some changes to Dovecot configurations network-wide to help prevent this from causing issues for any other servers until the core issue is fixed.

 

I just wanted to provide some details as to what has been going on with the S3 server as it has had some intermittent and short-lived downtime over the last few days that we have been working to track down. Thankfully we were able to trace it tonight and ideally assuming there are no other causes - the server should be rock solid stable again.

 

We're definitely keeping an eye on it .