I've always have to separate anxiety from like/dislike before I can decide; I can usually tell right away whether I dislike someone and I don't think I could every do very well if I actually dislike their mannerisms or things they say; it would be too distracting, like trying to reading a boring or too unpleasant book. But that's different from whether it's just "difficult." Mostly I look for the "truth" of what they say, whether they're actually listening and trying to get to know me, whether they paraphrase what little I might say correctly, little things like that. So, I'd know in one or two sessions I think whether it was worth continuing on for more.