It’s how you feel that’s important. Does it bother you enough to be a deal breaker on its own, or do you want to find more reasons (not) to keep seeing him, because you want more solid confirmation that he’s (not) right for you? Listen to your gut on this one.