Even if he didn't know that you might be sensitive to him saying that, there's surely a more tactful way to address your texts than by saying it was "annoying". That would bug me too.