I think you have to decide that the other side "doesn't get it" even when it looks like they have. I don't think they are clever enough to know when you are down and renew the attack, etc. I think they are trying to cross the boundary any way/time they can.

Can you think of some "stronger" ways you have enforced before and make them "automatic" and not needing "energy" from you? Push a button and the gate closes, tank trap comes up, etc.? Some of the problem, I think, sounds like it comes from your perceptions of yourself and them? Go for "real" and don't be nice about it. Snap at them! If you've told them before and they cross, use the automatic anger and don't be "shy" about it.

Yesterday I was visiting the grandchildren and my grandson (1-1/2) bit his mother. That's a no-no and he was strongly reprimanded, "No!" and given a serious time-out (my daughter-in-law left "him"/the room). No one smiled at his tears, etc. Your family crossing your borders as adults; they're not getting it! They are acting like children and get treated that way! Remove yourself (as opposed to them?). Grab your nephew and retreat to your car/go elsewhere with him where you can't be "reached" for awhile; library, fun eating place, etc.?

Don't be held hostage, for anyone. One person cannot be there for another all the time, no matter how much they might want to, not for a spouse, child, anyone! One's priority in life has to be one's self because without one's self in good shape, there can be no others to be helped. I try to remember the old airline admonition to get the oxygen mask on the adults first and then help the children with theirs. Figure out beforehand a plan to pick your battles? Always remember though that you have the emotional "brain" -- you know what's happening and can "plan" for it, the other "side" is just reacting, not responding. I love thinking about the difference between those two words and working to learn to "respond" to situations, not merely react. They're "blind" but you don't have to be because you're smarter about the situation in all ways.