He is a coward. I would try to detach from him, applying concerted efforts towards detaching yourself. If that fails, then, well, we would conclude that this is your lot in life, but you will have tried, at least.

Best!