I have a couple of ideas (which may or may not be useful - I am not saying they are the answer)
1. Husband's being annoyed is his problem. If he does not tell you when you ask, then what good does it to worry about it. He has made a choice - let him deal with it.

2. I can't tell if you directly said "I have to work - you have to take care of D and get her out of the house" if you didn't - then perhaps that directness might help (it would help me -if someone said "I have to get some work done today" I would not then go to that meaning "And because I am working you have to do X" - my natural response would be "okay do some work then" and then I would go off and do my own thing)

3. D is, even with her disabilities as described, in my opinion (for what it is worth) capable of being told - mom is working so you have to sit and play quietly for an hour. Set her up in the room next to you or her room or in the room with you - but with the direction that she cannot get in your lap or talk to you for X (reasonable) amount of time. I don't think parents have to be interacting with their children all the time - even if child is being whiny or clingy (something like - a hug and a kiss but mom has to eat her breakfast so you have to sit in your own chair right now)