Typically, sex in dreams does not literally translate to sexual desire. Instead, the occurence of sexual activity relates to a subconscious desire to connect with the other person in the dream sequence. Because the rule of thumbs with characters in dreams is "all characters are at their core a perspective of the dreamer", the second party in the dream must in some way relate to how the dreamer sees him/herself. The question is, what about your father and brother do you see in yourself? This can be both positive, as well as negative traits. These connections are most likely the source of a need to connect - the part of yourself being neglected or ignored.

In the waking world, it is likely that the attraction is a domino effect from recalling the dreams. You connect the physical excitement created in the imagery to your father and brother as they were the focus of the dreams.