I think it is good that you want to speak up with your manager, especially if he finds it difficult to talk to women. It will give him the opportunity to let you know that things are ok since you are opening the door for him to talk to you. Good job.