I think his comment “this is why we can’t be together” was inconsiderate; he makes light of what he has good reason to think is a serious concern of yours.