I think I would let him go especially as his criticisms are disturbing your sleep. Also if he is defensive when you criticise him.... I wonder if you could give some more examples. He sounds like a former friend (?) of mine. I am also an introvert and the former acquaintance (this person was too mean for me to think of them as a friend) was a little more of an extrovert. Are his criticisms ever constructive or always destructive? This person I used to associate with was always destructive with his criticism.