Exactly as others have said, you need to break up with him. There are lots of nice and decent people out there (aren't I just an optimist today...) but he clearly just doesn't fit for you and there is no reason for you to stay in a relationship you don't want to be in.

Maybe you can set him up with your mom.