I think you should consider carefully whether his refusal to hug you physically really is a rejection, or whether it only feels like a rejection.

The first is a deal breaker; the second is appropriate to explore within therapy.