I can feel pretty comfortable but "pretty" doesn't mean "REALLY" which means there is something thete that is keeping it from feeling "REALLY". Sometimes it takes people a little time to translate feelings into words & sometimes they just feel tgat walking away is better than the possible confrontation the words could cause.
Could be that the lightbulb came on for him thst the situation you were heading toward was not what he wanted to bring to where he works & a verbal discussion would have caused even more awkwardness in the work place not knowing how you might respond. Seriously the best idea was just to walk away & go back to how things were before.
I have had times when my view & how I finally saw things in a right way for me wete not accepted by the other person & were argued about because that was not what they wanted. When you don't know what someone's reaction will be it is usually best to just walk away & not create one