I think that golden-eve may be on to something. One possibility as to why this is on your mind is that you feel guilty for not being candid about why you asked to have him removed from your case. Presumably your calling his supervisor and asking for his removal was why he was distant. And now that he is leaving, your window of opportunity to apologize, if that is appropriate, is closing.

Do you feel guilty? If so, how much?