Honestly he can’t hold you back from meeting other people. He doesn’t have the kind of power.

Is this man visiting you in your house and taking you places? Are you ever going anywhere or all of this takes place in his house? You can stop going to his house, stop sleeping there and stop cooking for him and his buddies. Then it will be over. It’s been going on for a long time and the only reason it’s going on is because you keep going to his house. See what happens if you stop.

PS you are referring to him as a “friend”. How is he a friend? People aren’t upset when their “friends” have romantic encounters with other people. So he is someone you want more from and he isn’t giving it to you. It’s not a “friendship”. It upsets and stresses you and friendship supposed to be uplift you and make you happy