I honestly don't know if you should or not be with him. The only thing I can tell you is I started having doubts about marrying my fiance but he insisted that I marry him and I did because I thought I "should". In retrospect I wish I had waited longer to see how the relationship progressed. The relationship didn't go well for me but I do have a son I love dearly. But he is not a good father. So I am probably not helping at all, am I? I dunno. What is your gut feeling telling you?