I would say it is perfectly normal...my guess is that you feel more guilty about it based on the genetic ties vs. disliking some other random person...which you basically say here.
Do you have to see the family member a lot? If so, that probably makes it harder.