Background:
Last September I let the man (I'll call him CA) I was seeing move into my home shortly after we began seeing each other. I have a responsible job and have a good relationship with my sons (19 and 10). My 10 year old gets along well with this man. We mostly go fishing and watch movies together.

My older son got very angry at me for letting CA move in so quickly. He didn't feel comfortable at my house so he started spending most of his time at his father's house a couple of miles away and attends a local college. However, my 19 year old and I still have a good relationship.
He and CA don't seem to be able to find common ground. I feel very nervous when they are together. They are not hostile--but there is underlying tension. My counselor says it seems like they are trying to make me choose between them.

CA lost his job--his daughter and my son both say he is using me. However, he shows that he loves and cares about me. He rarely asks me for things; however, I am currently supporting him, mainly groceries and a place to live. He spends most of his time trying to get adjusted to his medications. But his "love language" is definitely spending quality time together.

(He takes medication and says that he is ADHD and bipolar. To me he seems obsessive-compulsive more than bipolar, but I am not the expert to say so.)

Here is my dilemma:
I told CA that I want to take my sons to the beach next week for a couple of days. I can't afford a two bedroom place; so I've rented one room with a double bed. In the last year, I have taken CA to the nearby beach, Key West and Orlando without my kids.
Now, he is hurt that I am not taking him with us next week. He says that he feels I am separating him from my life and my family. Whereas, he says that he wants me to be included in his family--which is fairly true in his actions.
Am I being inconsiderate and how should I address this with him?

Your insights are appreciated.