You are not wrong...... this is abuse.

If I may ask? - is this person you talk of over 18 years old and if yes, then why not have an adult make him leave the house via telling him or with the support of your local police department?

And as far as IF and WHEN you move out - you will probably be better off, for he will not be able to harass you just because he feels like it.