I think you made the right move by canceling the dinner. And it sounds like you are not a compatible couple and there have been past problems, as you said. Kudos to you to break it off.

I'm familiar with narcissists -my ex-husband and my brother. I can understand how a person can get unknowingly wrapped around their finger, be manipulated, etc. It's an awful thing but it sounds like your ex-boyfriend had that unfortunate dynamic with his sister. There is no room for a healthy relationship with you and him, in my opinion. He would need counseling, accepting who his sister really is, and creating extremely firm boundaries to protect your relationship and, his own self, IMO. I handled my situation by divorcing my narcissistic husband after many years and went "no contact" with my brother two years ago.

Stay strong and continue believing in yourself. =]