I am finishing my PhD in psychology so I have a lot of experience with people dealing with this type of problem. The statement below is based off of the assumption that you enjoy the company of your coworkers.
Sounds like you may have social anxiety disorder, and may suffer from agoraphobia (fear of certain situations). One of the best treatments for this type of disorder confronting and conquering. Meaning, do not shy away from uncomfortable situations, place yourself in them and figure out ways to deal with them effectively. For example, if you start to get nervous try to redirect you thought process and tell yourself that you are thankful and relieved to have friends and the opportunity to enjoy dinner with them. Try to make the most out of all situations and live life to the fullest.

Good luck