in your situation, no, i wouldn't change. i think circumstances and type of relationship make all the difference. my H rarely makes any demands on me and so if he felt uncomfortable with a doctor or even my t, i'd switch (and i have made that offer before with regards to my male t because my h once admitted to feeling a bit jealous but he recognized that having a t i connected to was important and he trusted me and the system).

however, it doesn't sound like your h would see this as proof you care. from just what i see here, it sounds like he would see it as proof that he can control you. in your situation i wouldn't change (tho i might consider switching to a t that has more time?).