does his kids live with him or are they older in college or something? I could see not wanting to live with someone with someone else's children involved.
I agree with the others--don't complain to him. in fact, feel free to come here and vent. yes, you *should* be able to vent to him, but it is mixing the message to him when he thinks he sees a simple solution.
if you truly go it alone or continue seeing him you have my support, doggie.