Yes, it may be uncomfortable but truth is...you are already uncomfortable.

Perhaps, discussing it with him will ease your mind? His reasons may be benign and it may have nothing to do with you.