Why would you lose him if you tell him to stop? Those aren't the only two choices. There should be a middle ground here, where you say, "Hey I don't like that" and he stops doing it but remains in your life. If he really would abandon his daughter because you won't let him do whatever he wants to you, then yes, this sounds abusive (at least emotionally). Even if being alone is painful it is better than being abused.