I thought about it, there is an another option... a lot of these situations happen in front of people. I don't know if this has anything to do with it.

I've managed to be assertive when it comes to my father, but there was a situation when another person was there. My father said something, this other person laughed and I ended up not saying anything.

Edit: Maybe these things bothered me, but I didn't know it. What it's called ... Depersonalization ? Or simply anxiety.

It's like I have a plan on how I want to be treated by family... I am not going to take ***** from them, but not a plan about how I want to be treated by anyone else, so I default to being nice.