He has cheated on you in the past; and does not want you to bring it up...manipulation of a hurtful situation that the two of you should have talked out.

He's also getting a kick out of making you jealous about "hot other chicks". Dump this guy, he will always find your weak spots and manipulate you with them. Hypothetically does not mean squat, he's intentionally aggravating you...don't let him have that kind of power.