When you've left, and then he pursued, sounds like a cat and mouse game. And you are a human being, with valid feelings and needs. Not a pawn in someone elses baggage.