Maybe hubby and you can talk to a professional who will help you come up with a reasonable solution based on your situation. (eg: your son's abilities, state of mind, needs, strengths, weaknesses; your own needs, etc...community resources...).
Then you can both sit down with your son and present the plan. And stick to it. Whatever it may be.
It is hard for me to remember now, (with my son's son asleep at my feet, and my son taking over the readying of my house for sale as I move on with my life...he is a healthy, mature man now...) but when my son was in his early twenties, I had to give him a deadline for either paying rent, or moving out. I knew he could live on his own, he had before, and I knew he was resourceful when he needed to be...he accused me of being willing to make him homeless etc----but, within the given time, he got a job, an apt., a girlfriend, and moved on; not without bumps and bruises, but it all worked out the best for all; he was able to feel like an independent person who could make his own decisions, and I had what I needed.
What helped me was being able to know his abilities, to express confidence in those abilities and to stick to my plan. He knew I wasn't going to give in, and he now says that what I did needed to be done.