Could it be that she was just a friend and he didn't want you to get mad at him for just talking to her and seeing her for lunch, etc.

If there was no sex involved it wasn't an affair, it was a friendship. I don't believe in "emotional affairs". It is just another word for jealousy