It sounds like these would be good things to talk over with a therapist. You say you have a "case worker" (I don't know what her role is) but it doesn't sound like she is very professional about her job if she talks about other derogatively. Do you have a therapist?