I don't think you are being inconsiderate and I think CA is being inconsiderate to put himself in the middle of you and your sons family bonding experience. He is not part of your family! He is not contributing anything to the family upkeep and an argument could be made that he is causing splits, as your older son and his daughter do not like the arrangement.

You do not have to make excuses or "buy" this man's approval of what you do, he has not seen fit to do anything to be a contributing member of your household so it is a bit unreasonable of him to ask for "benefits". If he wants to go to the beach with you, ask him to pay his way, to contribute so you can get a two bedroom place; he's not one of your sons, he's a grown man!