Maybe it's his unavailability and aloofness which makes him appealing to you? when I was very young I went through a phase of crushes on very unusual men the one thing they all had incommon was they were unavailable (in my case, unavailable = I was safe). Idk if this is relevant to you.

I'm wondering if there might be something you can learn/develop about yourself from this.