Kind of on the same idea as the rubberbands, but gentler: Maybe you could use a pair of those fabric covered hair scrungies. Use one on each arm to hold each sleeve away from your wrist. Like you, I wouldn't want to get my sleeves wet, and I would want to wash my hands thoroughly.

Mainly, I wanted to commiserate with your dilemma. OCD seems to always come up with a twist to every problem that is hard to keep up with. I know because I've had similar problems all my life.