I think you should start grad school at a time you believe is best.

When it comes to one's own academic and career aspirations, the only person who knows what you can and can't handle is you. When I was an undergrad, I had a lot of pressure to go to grad school right away or else I might never go. It turns out I didn't want to go to grad school, changed my mind about my program, and now am back in school for a different undergrad degree. I like what I'm doing now. Make the decisions best for you because you never know where life can take you.