I can kind of relate. For me (bio female, genderfluid), I can be comfortable with gay men, but straight men scare me. Even if I know him, there's still a small fear there. I think, for me, it's the testosterone and aggression that can happen. Gay men seem more docile to me, idk.