I struggle with similar anxiety when it comes to setting boundaries. You're changing the dynamic, and other people generally don't like that- there's a sense of control and comfort that's lost for them when someone else starts setting healthy boundaries. It's also hard for us to change what's been in place for so long. Hopefully, in time, it gets better and easier.