You write that you should have felt comfortable doing what your ex wanted. I don't agree, I think is perfectly fine for you to not want to do something, for any reason or no reason at all.

Having challenges with anxiety doesn't mean that you can't also have likes and dislikes, preferences, instincts, and the ability to make choices. Not wanting to do something isn't always caused by your anxiety - you can have both at the same time.

Are you getting help with anxiety? Therapy?