Has he ever cross the line with someone else, at least, as far as you know?
That would be a serious dealbreaker for almost anyone, unless they are of like minds.
Have you had a deliberate heart-to-heart with your BF to tell him that it is making you uncomfortable to feel like you are sharing him with others?
That should be the next step, I would hope.