I think if it bothers you then it is an issue. Lack of attraction wouldn’t be a problem for everyone (separate topic) but it is for you and that’s all that counts. He sounds like he’s a good friend though?