"my husband, he immediately gets anxious about it "we can't afford that, you'll get bored" etc. and basically refuses to let me quit my job. I guess it really doesn't help that he makes about $10,000 less than I do and that he absolutely hates his job and feels like he can't get anything better. PLUS, we have quite a bit of debt from various things (mostly student loans and car payment, plus a few credit cards)."
Your question is, are you being selfish? How about this; pay off ALL your outstanding debts, and find a way to live within the means that your H can provide. Can you do that? It is the responsible thing to do.
It is asking for trouble in your marriage though if he resents this. Just sayin' ...