Time to cut your losses. You deserve better, you really do!!! He sounds like he is playing you a bit, especially if he makes you feel crazy when you try and talk to him about it. IF you have been intimate with him, then it is a possibility that he is using you and maybe even dating other women or is married.
So sorry this happened to you. You are valuable and you need to find someone who will value you. Big hug.