I agree with what EnglishTeacher said. Why not take him to dinner, just the two of you, at least one night when he's with you? Perhaps a movie and a game (I know around here that the farm teams for the big leagues usually have pretty cheap games, around $12 bucks a ticket? 15?) Offer to take him to do things he likes to do that maybe he doesn't get to do much during the week. Perhaps encourage your husband to do the same with his kids while you're out with your son.

Good luck,
Ro