Maybe living AND working with someone like that would be hard ... but one of the two (just working with him) may indeed be ok.

good luck and it sounds like you have made many positive changes