He sounds like someone I wouldn’t want to vacation with even if it was 100% paid. He sounds like someone you should run away from and fast.

You don’t need his permission to end it. Since you two don’t live together and have no kids, break up should be fast and easy. Then you can build a beautiful life without this loser. You can do it. Good luck

PS Him demanding you go on vacation while unemployed is absolutely insane.