I'm not sure I agree completely. Shirking responsibility leads to (perfectly reasonable) guilt, which is not conducive to happiness. Not having responsibilities in the first place is maybe a happy state, and I do avoid some responsibilities (it's one of the many reasons why I have never wanted children). But having responsibilities and being capable of fulfilling them is also a positive thing. It's the state of feeling unable to meet one's responsibilities that is miserable.
I'm also not sure that kids and elderly people are all that happy....