I think you need to realize that you did the best you could when he struggled and that you did have challenges at the time. You are probably hanging on to guilt and self blaming when you shouldn't but instead did the best you could at the time.