I set up slimta as a mail relay. I confirmed that mattermost opens a concurrent connection for each message (over 200 connections for "all" notifications). Configuring slimta to queue them up and relay them over a max of 5 connections to the actual smtp server seems to work very well.
An added bonus is that slimta will retry failed emails (occasionally there's a timeout or "Insufficient system resources" error on the smtp server).
I hope this is a useful tip for anyone else who gets the "maximum number of concurrent connections" error.
@jwilander I guess it would be nice if mattermost supported a max smtp connection count and retries itself?