I would try to tone it down a bit, but I don't think it is an unforgivable problem. People communicate a lot: it's what we do.

As you said, academia is special in that way, being more of a community of sorts, but it is also very easy to waste time (your own and his). He might not have a problem with, seeing it as an escape of sorts, but he also should be protected against himself.

Try to stay responsible at all times.