I happen to agree with Skeezyks, I think he was simply offering up a rationale for why you are receiving harassment, I admit without realizing what he was doing. I also doubt that there was anything clandestine about it.
As ruh roh said, if you're otherwise happy with his behavior, I would probably give him the benefit of the doubt, but as everyone seems to agree, definitely discuss it with him.