If you are concerned, you should probably discuss this with him. Don't hold back. It's importan to get it all out on the table. Then see what happens. If you are not comfortable, don't do what makes you feel like that.