To me, the age difference isn't so important as the stress you're dealing with. It's time to sit down and talk with him. Explain everything as you did here, see if the two of you can come up with a good solution. If you can't find a good solution, or he won't listen to you, then it's time to move on.