Time to let go?

Hi there! So I guess I'm looking for some advice since I don't have many people to ask.

My life had been on a downward spiral since my dad died 5 years ago. Long story short after an 11 year relationship ended I was single for about 6 months and then came across my current boyfriend. We will be together 3 years in April and I feel like we have a great connection...but with MAJOR issues. Sometimes I think I'm holding on because I have no family or friends really. I guess I want to get advice on if this issues are relevant or am just being selfish.

He has never told me he loves me. I try not to let it get to me...as I believe love is more of an action than a word BUT it is nice to hear. He recently told me that he associates "I love you" with being trapped due his past bad relationships...I'm thinking that is a commitment issue?? Then there's the intimacy issue...It use to be amazing. Holding each other and talking about our dreams...now...nothing. He goes to sleep at the drop of a dime and the only time I get him to even put his arm around me is when I get pissed. I've tried to talk to him about it and the only thing thing I have got from him is he depressed or aggitated. I try to help him...talk to him...anything I can do to make him feel better...but the same thing. It's like he doesn't care and nothing matters.

This is EXTREMELY hard for me. Because I feel like I'm barely keeping myself together sometimes and I get NO support from him. I feel like I'm holding up 2 people on my own. Should I not give up on him or leave and try to make myself better?? Just one of the many questions I'm faced with...