I think you have no reason to feel guilty at all (and yes, he's probably guilt-tripping you..). Move on with your life, you deserve better..