Yes you have every legal right to kick his *** out. Call the cops and tell them he refuses to leave and they will come by and physically remove him from there and possibly take him to jail if you go into the other things about him pushing you etc...

If you don't want to go to those lengths quite yet then say "if I come by and you're still there I WILL call the cops". then, if he still doesnt leave, follow through with your threat. That's what police officers are there for!