Hmm... I think the word "affair" definitely does not apply because it is still more or less reserved for extramarital affairs, and you are not married.

I would not talk it over with him anymore as that would feel draining to you and not yield productive results. Tell him instead that you can agree to disagree. State your position and let him have his. Then, tell him that he is entitled to his position, but should keep it to himself. That should stop verbal accusations. If he cannot stop himself from making verbal accusations, leave him altogether. I think he is prone to having pity parties...