My advice would be to dump him. Why waste your precious time on this person? I felt the exact same thing with my ex but let it slide. Soon after I got dumped cruelly. It's not worth it just to stay with them.