How long into the relationship did you start seeing his real colors (which are concerning, in my opinion)?
Also, i think you know the answer to the question. Listen to your guy feeling because it smells bad.
It might impact your self esteem to be in such an intimate relationship with him.