How do you know there is physical attraction when you have never been physically with him or he with you. Until you actually meet, it's all just in the mind & thinking the attraction will be there.
I think you are just being wise to doubt anything about the relationship at this point other than you communicate well together long distance. Go with your GUT feeling. I learned the hard way IRL, not long distance, to ALWAYS listen to one's gut feeling. I had one that told me not to get married to the guy I got married to. 33 years later I left him for exactly the reasons my GUT (intuition) told me not to marry him in the first place. I had tolerated what I didn't like way too long. There is something about him that your intuition is aware of....LISTEN TO IT!!!!