I don't know what outcome you would like. If you cannot and are not caring for your son most of the time (during the week), then your influence has to, by default, lessen. You could go to a lawyer and straighten it out but, with your ex-husband caring for your son most of the time, I don't know what would change? No one can/will interfere if your ex calling you names and verbally bullying you; as the social worker says, he's "allowed" to say whatever he feels. You broke up because you could not get along and nothing has changed there.

What do you want? You can't really suggest that your ex- take care of your son most of the time but keep his mouth shut and you still have the greatest influence, that won't work? If he has not been diagnosed with a mental illness that would make him a real threat to your son (who has not the ability or experience yet to "fight back" for himself or state how he feels and set boundaries that you do) then I don't know what use legal could have. If you have a T of your own, I would discuss the situation with them.

Someone has to take care of your son; if you cannot, it sounds like your ex- or social services has to and when either of those have care, they get it all the way during that time, you don't get to control from the sidelines.