I agree with the other user. The only way to find out whether it is your business is to ask him. Besides, it's also possible that the person could lie rather than be frank, if you're afraid of asking the question.

Anyways, let us know his response if you do ask. It seems rather strange. Reminds me of some boss of something sketchy having his cronies around. Lol. Not saying that's what it is, of course.

Is there anything more you could tell us about your date and the group of guys?