You don’t need anyone’s permission to end a relationship. Most certainly he can’t “not let you”. Him not wanting it is irrelevant. Yes when you leave a relationship, it’s likely other person wants you to stay. But it’s not up to them