not feeling up to posting much but this needs to be said
IT WAS NOT YOUR FAULT.
I would stay away from him, tell someone you trust if you need to enforce it.

He is taking advantage of you in that situation and you don't deserve that.