Has he ever cross the line with someone else, at least, as far as you know?

That would be a serious dealbreaker for almost anyone, unless they are of like minds.

Have you had a deliberate heart-to-heart with your BF to tell him that it is making you uncomfortable to feel like you are sharing him with others?

That should be the next step, I would hope.