I think the main issue isn't about sex or whatever you describe as intimacy. You hit the nail on the head when you said he avoids conversation about it at all costs. That IS him pulling away from you and intimacy with you.

I recommend reading a book called "Too good to leave, too bad to stay" by Mira Kirschenbaum.

It sounds like you are in what she calls a state of relationship ambivalence. You don't want to leave but nothing is really changing to make it worth your while to stay. Him flat out refusing to talk about something you feel strongly about is what she calls 'off the table-itis' because whatever you want, he takes off the table with no way of getting that need met. In your case it's not only communication and talking but also physical intimacy. She calls that as one major reason to leave a relationship because if you can't talk you are not getting your needs met. That is disrespectful of him and him being emotionally and physically cold.