I'm so sorry you're feeling this way! Can you try journaling it all out and then ripping it to shreds? Sometimes that helps me get some of my pent up emotions out. I can understand how you're feeling -- my mind would jump to conclusions too. I hope you are able to have a calm conversation with him and he can put your worries to rest.
Sending you lots and lots of warm thoughts and hugs!!