We can't always understand why we react the way we do, Googley. Something about him triggers fear in you, for whatever reason. What's important is finding ways to manage that, and talking to him in general terms about your need for accommodations might be one way to help manage that fear reaction.

You're right that he might react in a stigmatizing way. What will you lose if he does that? Is there a way that you could seek another adviser, in that case? Similarly, what will you lose if you don't ask for these accommodations? Perhaps comparing the possible outcomes and how you'll deal with them can help you decide what to do.