You have every right to be suspicious or on edge. The fact he said you were on a break to another woman that he's texting back and forth would make me crazy too (well not a lady but a guy for me) but anyway..

It does sound pretty disrespectful at the very least but more seriously as if he's doing some pretty heavy flirting with her too it sounds like.

No you should not let this go. he's wrong.