I've also had this issue. We've set up GitLab Mattermost (within GitLab omnibus) with a general team for users to join. The main team is shown when a user first navigates, and prompts them to log in with GitLab. We then receive the same error posted in the OP.
[2016/02/17
10:19:00 EST] [EROR] /signup/gitlab/complete:SqlUserStore.GetByAuth
code=500 rid=xxtpz7rid784ukk4rg3zaojetr uid= ip=10.3.1.31 We couldn't
find an existing account matching your authentication type for this
team. This team may require an invite from the team owner to join.
[details: teamId=9bao1i7ptigbzna3wd7jjz9suw, authData=5,
authService=gitlab, sql: no rows in result set]
We haven't yet setup OAuth/LDAP yet, just default GitLab user accounts.
There seems to be a weird workaround. If the user first clicks "Create one now", they can then join the team normally.