At minimum it's an emotional affair. So he doesn't want you involved with anyone else emotionally. That's his line. Either accept that or leave and live the way you want.