You need a new case manager and your meetings should only be taking place in an office or over the phone. No coffee, no walks, no sharing of “problems” other than the reason they are your case manager. You are not their friend, they are not your friend.

Somehow the professional line has been blurred with your current case manager, so the whole working relationship is ruined. Start over and get one that is a professional.