Does he have a counselor? Maybe you can discuss it with someone and they can get an outside look at your relationship and figure out what gets in your way? My stepmother and I didn't get along, we couldn't communicate well, kept "missing" each others signals/meanings. Did you get along with both your parents? Why or why not/what kinds of problems did you have with either? Could be he unconsciously reminds you too much of yourself or you're afraid he will? I'd talk to a counselor, see if an outside observer can't help a bit. It doesn't really matter that he's your son; we just don't get along with everyone! He's your son and you love him but may not like him/his personality. Have you tried opposite behaviors; when you get upset, try laughing and hugging him instead of leaving him? I don't think he's deliberately getting under your skin (yet :-) but maybe he is, maybe you only interact with him when he's being a problem so he's becoming a problem more often which causes you to leave him which causes more problems, etc.