Yes, like InRealLife45 says I think it is less about what he is doing (I don't think what he is doing is unethical) but more about how you feel about it and your comfort/discomfort and so please discuss this with him in a way you think is appropriate (I know you are concerned about the rapport) because maybe he is thinking that he is showing you care by hugging you at certain times and I don't think he would be happy to know he is in fact making it worse for you. In other words, if he becomes aware of it, I'm sure he'll be happy to show his care in other ways that you find more comforting. And if he has other intentions, the discussion will also make him aware that you don't like that and that you will speak up about it.