I think your t is right. It isn't your fault that you feel the way you do. I think that you should be honest with your H about it though, he deserves to know before you act on them.