How about saying, "No thank you." You shouldn't feel obligated to agree. I'm sure there is a reason for his inviting you, if you wish to not feel only like a guest, then why not ask him why he invites you without inviting a single guy for you to check out?

How would you like them to make you feel, btw? Do you want to help cook or set the table??? Do you want to sit and chat with the co worker? I don't really understand your situation other than you feel uncomfortable. That's enough for me, btw... to just say no.