This is exactly how my sex drive is. When manic my sex drive is insane and the woman I am with couldn't be happier. When I am down, it's rare that I want to even try. At first it was very confusing for her (and even a little to me) until I finally had a bipolar dx and then it made all the sense in the world.

Medications can also really effect sex drive (SSRI's just kill mine) but it sounds like your's is a mood issue where it comes and goes. That's exactly what it is for me. What my girlfriend has said is that now that she understands the ups and downs (which I hope to level with meds soon, I am currently not medicated) she knows that she will just enjoy when my drive is increased and not hold the lack of drive against me when I am down.

I think understanding that this is probably a mood thing which I bet a lot of BP people deal with should help it to make sense to you. In turn, you can reassure your husband it's not him and maybe he can understand better as well.