Tell him exactly what you've told us. There's nothing wrong with your feelings about the situation at all and I believe he'll understand.