To me it sounds like there is some reason for concern. If your children are saying he's doing things that hurt/intimidate/upset them, then take their concerns very seriously. In the chance he is doing these things without the intention of making them feel the way they do, then someone just needs to sit him down and explain that to him. Joking or not it's not okay.
But, he could be in the process of becoming more abusive. Sometimes people who abuse will intimidate the children of their partner to 'get rid of' them. I was in that situation, where my mothers boyfriend treated me like hell but manipulated my mother into thinking what he was doing was okay. My aunty is now in a very similar relationship.
He will convince you he was just joking or playing so you think it's okay, but keep intimidating your children until they leave, so he can 'have you to himself'. It won't always be obvious to you though because if it were, well he wouldn't get away with it would he? And I can guarantee you, if he is intentionally doing what they are saying he's doing, it WILL get worse.
I would suggest sitting down with your children without him anywhere around at all (even if he's in the same house but in another room or sitting out in the car it may may them too uncomfortable to be fully honest with you, when talking about these things is scary enough), and ask them to share with you what they experienced with him. Do NOT in any way make excuses for him while they are talking to you (i.e. 'oh he was just joking, he didn't mean it like that'), because it lessens the validation of what they are trying to say. Just let them tell you what he did, ask them how they feel about him and the situations they've had with him, and then make decisions about what you'll do about it when they aren't around. Children do know more than people realise, but at the same time a lot of how they feel and the way it affects them may not register to them for a very long time, even years. They aren't bad or wrong for feeling intimidated, intentional or not.

Hope that helps somewhat.