My boyfriend is the exact same. We've been together three years and we hardly have sex. I too have worried endlessly that it was me and felt awful, but honestly if you're still together as a couple, it's not you.
In my case my boyfriend has gained a good amount of weight since we were first together, he is slower and uncomfortable in his body so that's why we don't have sex. I accept him for that and understand because previously I had the same issue.
The confidence thing is big for men in sex and if he says that's why then chances are that's probably exactly what it is. Don't go trying to change all these things about you, after four years it's clear he loves you for you.