I think you have to speak up for yourself, and establishing your turf is important in this situation. If you prefer to speak silent or do not wish to mingle with other people, this abstention does not imply that someone on their own starts calling shots for you. So you have to be vocal here by clearly stating that whenever you have to make a decision, no one else needs to step in. If you'd need any help or advice, you will expressly consult these self-styled advisers...