I have a rule for myself, that I won't fantasize about someone who is currently in my life-- so as not to encourage any infatuation that could go down the wrong path. That is more of a practical issue than an ethical issue for me.

I don't think its wrong to fantasize about someone other than your partner. I assume that my boyfriend sometimes fantasizes about people other than me, and it does not bother me. Studies also show that the vast majority of people in long-term relationships fantasize about people other than their partners. What you are doing is normal, and in my books, there's nothing wrong with it.