>I still hope he never brings it up again.

Can you tell him not to? I would, in no uncertain terms.

Edit: I wrote the above a little w hile ago, but now I feel like it was a bit pushy to say that.
It sounds like you're dealing with it and you want to be more assertive and stand your ground. And I hope things improve for you and you keep getting what you need in your support sites.