I agree the fit is most important. One way I differentiated is before I was a psychologist, they were clients, afterwards once I became a doctor, they became patients. If I had to refer to someone anonymously for some reason, who was also a friend that received "advice" from me from time to time, then they also were referred to as "client."

However, if someone wanted to discuss this and didn't like either term, then of course it's only proper to not use it.

BTW I do see a PhD Clinical Psychologist, and I am his patient.
TC