I wouldn't think it would be wrong of you to ask. I wonder why he feels that way? Like you said it is just a basic letter stating that you are seen there. I don't think he should be uncomfortable with it. That's just my opinion anyways. Hope things work out for you.