Maybe you feel like his sex life is more real when it's in the house? When he's out the house you're not 100% sure what he's doing and therefore can't feel jealous.

I also feel discomfort when I'm next door to people who are having sex because I'm not a voyeur. Is there some way you can go spend the evening with a friend or an acquaintance when he is doing this? Or maybe ask if he can go with his date to their home?