I think jmo351 hit it on the head. Perhaps he is simply taking your lack of desire for him as a sort of rejection. Hard to say from where I sit, but that sounds likely to me. I also think he should be more understanding of your lack of sexual appetite since you certainly have a lot on your plate. Obviously you are not a bad person for not wanting to sleep with him at times. I'm glad you communicated to him that sex is not the most important thing for you in a relationship.

Good Luck.