I don't have any kind of disorder, and neither does my brother, but we often have very dilated pupils. Sometimes people comment on it, but fortunately no-one makes a big deal of it. Even my eye doctor last time I was there remarked on how he didn't need to use drops to dilate my pupils that day since they were already large. But he didn't think anything of it.
I don't know what to tell you about how to get the people in your life to calm down...but at least you know now that there are people who just have dilated pupils, for no particular reason!