What an awful, awful (valid) feeling.
I can definitely picture the whole thing from reading the post. I think if you want to continue, this would be something you might have to get used to experiencing over and over again. I don't think it was an accident that you reached out for the first time and he indicated he doesn't go there.
There are many points, but this is not one of them. It has nothing to do with boundaries. Boundaries are for actions, and what you're describing is more of a feeling.
I really need to stop procrastinating and get back to work but that is my 2 cents.