I was thinking that if they were married and or had children, they need to be at home for the dinner hour, whereas the time during the day is their own.

Also, i wouldnt call it "setting boundaries". If everybody in the group ordinarily gets together during the day, then yeah thats the group boundary. Find a different group that meets at night if thats your preference. Your boundary starts with yourself.