Matt
I have been in your shoes before. My fiance at the time thought that I didnt love her because I wasn't showing her enough affection. I reassured her that I still loved her, but it was not enough. I told her it was me, because she was having doubts that I was not attracted to her anymore. I thought about it more and more and I found the reason I was not affectionate as much as she wanted. I worked on it and she worked on what made my sex drive go down some, but it was not enough. I say look in yourself to see why you are the way you are.
i found out it was because my parents were not affectionate to each other. Also, my spouse disappointed me by promising things and not doing them. I didnt know it but it affected my sex drive and it drove us apart. Stop the wedge my friend.