I heard that same sort of thing when I was in college. It's like some people think doing a graduate degree isn't work - they probably have no idea how many hours you put in.

When I was in college, sometimes I'd get the opportunity to teach a lab or a weekly class study session or even grade papers. It wasn't many hours and it didn't pay much at all, but it was kind of nice to do, and it made my parents feel a little better.

To answer your question, though, no, I absolutely don't think it's wrong not to have a job while in school.