First of all I want to say how strong and special you are to stay with him. Not everyone would be as nearly supportive as you are.
Keep on taking deep breaths hun....and use as many resources as you can. I am assuming that his family is supportive? Maybe you could really talk to them about his past and have them help you for the future.
If nothing else this place is here for you.

Jessica