You feel bad...but you are able to say no.

Maybe that same skill and strength can be used when he is guilting you and annoying you?