I don't think you have anything to worry about Hamster. I personally am not attracted to African-American women, but I have close friends who're black women. They're good friends. I just don't see them in that light.
Everyone has their preferences. Nothing to be bothered by.