You are right to realize that it is your behavior that is causing the problem. Maybe you could start by deciding to keep your feelings to yourself except for once a week? Schedule a heart-to-heart time every Thursday at 9:00 or something, while you're out having pizza?

My husband and I enjoy some travelling, road trips and we are always in the car together. Sometimes I read aloud to him from my Nook and we discuss what I'm reading. Sometimes we just talk about whatever one or the other of us is interested in. You could talk in general about trust, etc., confess some of your worries/thoughts (about his being younger, etc.) and maybe learn things that bring you closer and ease your worries?