Is it possible for you to talk about your feelings with him? Specifically, the awkwardness and the guilt you're feeling? It sounds like it would be pretty uncomfortable to bring up the topic, but you'd probably feel even worse if you never get the chance to!