I can say from one man's perspective that being rough, hitting, spanking can be so emotionally draining (to the point of upsetting) that after a point he needs a break. I bet he enjoys it because you enjoy it, but at another level he probably also hates lifting a finger to you even if you want him to.

I agree with Healing4Me in that it sounds like it's becoming an issue. I would certainly suggest talking with a therapist so you can both work through it, negotiate what will work for you both and keep your intimacy.