Hi, Scone, and welcome to Psych Central!

In many relationships the sex is really great at first and then eventually cools down some.
I agree, though, that the distance is a problem. He doesn't see you for awhile, then all of a sudden he's with you, and you're expecting him to perform.
To be honest, it sounds like he is deciding whether he wants to continue the relationship. Long-distance relationships are tough to maintain in the best of circumstances.
If you and he do see each other again, then it might be best to just try to have fun/enjoyable times together, getting to know each other, and not push him for sex. Maybe over time he will respond again.
Those thoughts are what come to my mind off hand.