I agree with many of the above! This kind of behaviour is more than one red flag!! Get away! And keep in mind the parts were you said things like, you would rather not have let him in not started any self loughing.

Yes, it can make things easier when you know how your partner feels about you beeing in contact with an ex. People have different opinions about this and can avoid unnessary conflicts to talk about it. This is not to say, you did anything wrong! It is just my learning and they dont have to apply to anyone else.
And again, there is no excuse for this kind of aggression!

Stay save!