Thanks, yes they did not have this enabled but they had a different "@otherusername" in their NotifyProps. Resolving that restored normal behaviour.
This is still unexpected behaviour, right? With those settings, I would have expected them to simply not get a mention, and the "they do not belong to this channel" message should not be shown.
I agree that removing the setting sounds sensible!