I hope you do not mind me speaking up, but this line you wrote - for me it always screams 'manipulation'. I see and hear it quite often, and its like a woman has been worn down for so long that any tiny crumb a man offers is taken.
Well.. I do not think its good enough. I hope am not out of line here, but with his affairs and his nitpicking over your appearance, I do not think he respects you, and he doesn't seem to respect his own daughter either.

I hope you know that if you do decide to leave, it is not your fault, and nor should you care about the opinions of his family and friends. I think, after living this long with put-downs and 2 affairs, you can hold your head up high. You've worked harder than many people would've done.

Good luck in whatever you chose to do.