Just make sure you keep something dated and in writing detailing this to keep in your file cabinet. Just in case he carries this nasty reaction into the workplace.
You are coworkers. Sure some people do have workplace involvements, at the same time there are many that don't and it's even more presumptious on his part to react so strongly to your towing that line.