i have been in this same exact situation, to protect your visitation/co parenting rights to your son have it stipulated in the divorce papers the days of the month you have exclusive access, or you could be in for a nasty shock, verbal agreements mean nothing in childrens court and the father is normally the person who loses out in these cases,

i made the mistake of trusting her when she said i could see my son whenever i wanted, in reality i get to see him 3-4 times a year if i am lucky