Whether or not this father thing is true, I think you should speak to your boyfriend about it. If nothing else, it will help him to understand your insecurity and help you two through.

Although it is good to understand where your insecurities may come from, it is also important to remember that a reason is not an excuse, and that you can control your fears, if you wanted to.

Good luck.