Well... I don't know.

Yes, you do need to be mindful of the neighbors. At 11 PM at night they're probably trying to sleep. But beyond that, it seems to me you should be able to clean when you want to... or need to. From what you wrote, it sounds as though perhaps your hubby is a bit of a control freak?
I presume your husband has your best interest at heart, at least when it comes to what you're eating. But it is true, I think, that even the best of intentions can be overdone sometimes. It must be difficult being surrounded by people (your husband, your minister) who think they know what's best for you. Personally I think it can be better, sometimes, if a person just succeeds or doesn't on their own. At least that way, you know who to congratulate if you succeed... or blame if you don't...