You definitely are not the worst dad and the son will love you dearly, so no problem with that, but I feel that your wife should rethink the priorities. If you need a break, she needs to drop gardening and give you a break. Prioritizing gardening over your mental health is bad.