There is a major genetic component to autism and aspergers. However, it sounds like his family member with supposed aspergers hasn't even been formally diagnosed. Personally, I have several family members who fall on the spectrum, and it was a serious worry when my youngest sibling was born. She turned out to be the most gregarious of the bunch, though.

There is no way to predict these things. Even if there is no autism in either of your families, you might have a child with aspergers. This same possibility exists with many disorders your child could have. He/she could be blind/deaf, be born with down syndrome, develop asthma, get into a wreck and end up in a wheelchair, etc.

If you absolutely can not handle this possibility, then I would avoid having children. Adopting would increase your chances of getting the type of child you want, but it would not prevent serious injury or illness from disabling the child later in life.