Depending on what it is you want to do instead of being in the band you can find a group that does this activity, or a class, or a workshop or something. Tell your husband you will be going to this meeting and will not be available on whatever night(s) for the band. That way you can do whatever you would rather be doing while still being in the band. You can slowly withdraw and it won't seem like you are letting him down.

Be sure to remain supportive and listen when he talks about the band. Also, go and hear them play when you can. That way he will know you are still involved, just not to the degree you once were.