I would go for that look or sign of saying "yes" or "no" for you.

The sign for "sorry, No" could be grabbing his hand and wrapping it around you as to say I just want to be held - assuming that is comfortable for you...

I've always believed in personal communication in the form of sign or look - just never practiced it myself - wish I had...