Ophelia, practicing is nerve-wracking but important. Practicing and finding out that your worries are simply your worries and not necessarily what is real. It is complicated because it could be real, another could be upset with us. Then there is that to explore. What is the fear there... If someone is upset with me, then... ?

Keep practicing and I think you may discover that the likelihood of others being upset with you is much less than you expect, and that there is a range of responses you have available to respond with if others are upset.