It sounds like YOU had negative experiences with him and that's enough for you to find someone new.

As for the clammy skin comment, try not to assume and ask the right questions. I agree, it would have been professional to explain to you why he pointed this out.