I have a couple of ideas (which may or may not be useful - I am not saying they are the answer)
1. Husband's being annoyed is his problem. If he does not tell you when you ask, then what good does it to worry about it. He has made a choice - let him deal with it.

2. I can't tell if you directly said "I have to work - you have to take care of D" if you didn't - then perhaps that directness might help (it would help me -if someone said "I have to get some work done today" I would not then go to that meaning "And because I am working you have to do X")

3. D is, even with her disabilities as described, in my opinion (for what it is worth) capable of being told - mom is working so you have to sit and play quietly for an hour. Set her up in the room next to you or her room or in the room with you - but with the direction that she cannot get in your lap or talk to you for X (reasonable) amount of time.