I wouldn't stress it Bamster. You'll have certain people that you're naturally attracted to physically and then there will be other people that the attraction will come with interaction. It's not a conscious preference.

In my case, I'm a butt man. It's not that I make a conscious decision not to be attracted to girls with small booties, just something that I find attractive.

Are you instantly attracted to every single Caucasian guy you meet? (Hehe, doubtful.) So why would skin tone be any different? And as you found, there are people of other ethnicities you'll find attractive at times, so don't put pressure on yourself. It's too easy to start beating yourself up over something like this.