I wonder if the anger is really about a dependency on you...he clearly wants a place that is his own but you pay the mortgage. Why? He may really need to stand on his own, be apart from you for a time. The anger with his own dependency may be turned on you.
Don't know. I have adult children and I wouldn't be at their homes without an invite or if I happened to be in the area & let them know. They are in charge of their lives. I would suggest you talk to a counselor yourself.