You're not being a party pooper in my opinion.
I don't know what its like to have any children but I think you're doing the right thing. Taking time for yourself is fantastic and I highly recommend it, but you do have all of those responsibilities. I think if you can reach a healthy balance and take time for yourself and go out and enjoy yourself without feeling guilty that you really should.
Have you tried explaining to your friend that you need to take care of your children, but that you do enjoy going out - on occassion and not constantly? Or what about doing something enjoyable that you can do with your friend and the kids at the same time? That way you're enjoying yourself still and taking care of the children at the same time.
Just random suggestions, feel free to ignore me.
(((((((((desirae))))))))))

Take care of yourself and I hope you can get things sorted out with your friend.