I totally felt this way about my first h. Then i knew it was time to get a divorce when he was walking behind my car and i wanted to "accidentally" slip it into reverse. I think this could be a psychological fear of abandonment, because even after i wanted to run over my first h, i then became similarly attached to my second h, that is, really not in a good or healthy way.

You have written here about how sometimes you lose your temper with him. Maybe if you just generally, mindfully try to improve your relationship with him on a daily or hourly basis, it will build a stronger foundation and alleviate fears.