Assuming you would not be violating the terms of a restraining order, you have a right to access whatever areas of town you want. You do not have to make yourself invisible in order to make your ex-therapist comfortable.