I was friends with only one of my ex's. for a long time. Everyone has different boundaries and feelings about significant others and exes coming around. OP, if you are not comfortable with it, that is ok, and it's ok to be honest about it with your SO, too.