Am I excluding him from my family?

Background:
Last September I let the man (I'll call him CA) I was seeing move into my home shortly after we began seeing each other. I have a responsible job and have a good relationship with my sons (19 and 10). My 10 year old gets along well with this man. We mostly go fishing and watch movies together.

My older son got very angry at me for letting CA move in so quickly. He didn't feel comfortable at my house so he started spending most of his time at his father's house a couple of miles away and attends a local college. However, my 19 year old and I still have a good relationship.
He and CA don't seem to be able to find common ground. I feel very nervous when they are together. They are not hostile--but there is underlying tension. My counselor says it seems like they are trying to make me choose between them.

CA lost his job--his daughter and my son both say he is using me. However, he shows that he loves and cares about me. He rarely asks me for things; however, I am currently supporting him, mainly groceries and a place to live. He spends most of his time trying to get adjusted to his medications. But his "love language" is definitely spending quality time together.

(He takes medication and says that he is ADHD and bipolar. To me he seems obsessive-compulsive more than bipolar, but I am not the expert to say so.)

Here is my dilemma:
I told CA that I want to take my sons to the beach next week for a couple of days. I can't afford a two bedroom place; so I've rented one room with a double bed. In the last year, I have taken CA to the nearby beach, Key West and Orlando without my kids.
Now, he is hurt that I am not taking him with us next week. He says that he feels I am separating him from my life and my family. Whereas, he says that he wants me to be included in his family--which is fairly true in his actions.
Am I being inconsiderate and how should I address this with him?

Your insights are appreciated.

I don't think you are being inconsiderate and I think CA is being inconsiderate to put himself in the middle of you and your sons family bonding experience. He is not part of your family! He is not contributing anything to the family upkeep and an argument could be made that he is causing splits, as your older son and his daughter do not like the arrangement.

You do not have to make excuses or "buy" this man's approval of what you do, he has not seen fit to do anything to be a contributing member of your household so it is a bit unreasonable of him to ask for "benefits". If he wants to go to the beach with you, ask him to pay his way, to contribute so you can get a two bedroom place; he's not one of your sons, he's a grown man!

Thank you Perna. You hit the nail on the head.

I do feel like I have three sons. I have repeated my past marriage and my previous, serious relationship. But then codependency is an issue that brought me here.