Might mean the fear of being personally accused or attacked at such gatherings, or you are being hard on yourself. Feeling like you do not deserve a good time.