Bee, IMO, if your boyfriend doesn't want to talk about what got you in this position, he's avoiding not only the subject but the problems, too. An apology isn't as important as him coming to terms with what happened. Nothing will be straightened out nor changed if this doesn't happen. The fact that he got angry when you mentioned your family being afraid you're setting your self up for hurt again, says to me that he doesn't take ANY responsibility for what happened. There's just too many red flags popping up for me. You've already put you and your kids in harms way. It wouldn't be wise to do it again. Your kids need more stability than that!

Good luck!