WebRTC is working for us. I set up Mattermost according to the documentation using PostgreSQL.
I installed Janus using the unofficial Debian repository here: http://ag-projects.com/debian/
Janus/coturn were configured using the configuration files from the Vagrant image: https://github.com/mattermost/mattermost-webrtc
The following ports in the firewall need to be opened:
TCP 443 (HTTPS)
TCP/UDP 3478 (STUN)
TCP 5349 (TURN)
TCP 7089 (Janus)
TCP 8189 (Janus)
Initially we had the same error in Chrome (55) and Firefox (49), but the issue was that we are using self-signed SSL certificates for Janus. By adding our CA certificate to Chrome, we are able to make/receive WebRTC calls via Mattermost.
Ensure if you're using self-signed certificates, that you've created them using a CA and the CA certificate has been added to the browser. If you're using a certificate from LetsEncrypt or a commercial CA, it shouldn't be an issue.