I suppose there is a possibility that you might be finding him attractive because he is a "safe" subject for fantasy because you know you will not take it farther.

I could be way off on this, relationships are hardly my forte.