You sound like a parent that is feeling all the normal thoughts about their child. Why would you be abandoning him if it's HIS choice to visit his mum and if HE decides the grass is greener and wants to stay with his mum? Sounds to me like that would be his choice....not something you brought up and asked him to do.
The other thing that really stands out to me is that he is bullying you in order for him to get away with being a lazy slob. If you are the one who pays the rent, feeds his need, provides his room and gives support then I would say you have every right to expect him to do his share...and it doesn't sound to me like you are being unreasonable here in your expectations of him.
Regardless of whether or not you could hurt him when he lashes out at you, he is bullying you and you have every right to defend yourself and teach him some respect. Maybe you could find some help in your community as to how to deal with his outbursts and lack of respect. Many times it's a matter of the parent changing and the child follows suit.
As a mom, I certainly do understand that danged if you do and danged if you don't feeling. What I try to tell myself when I'm feeling that way is to follow my gut instincts. They never let me down and are always right on target!
Too bad kids didn't come with a manual eh? Wishing you luck and I hope you find some peace through this issue you are having.

sabby