As Hankster said, your first responsibility is to your children. If she isn't stable enough to handle a more extensive relationship, you have to do what you feel is best for them.

I have an uncle that is as crazy as a mad hatter. But the kids love him, although they do not understand him most of the time. He has a strange sense of humor. He'll say two unrelated words and laugh and laugh and laugh. The kids always thought it was fun. He was always laughing. Since they were the only relatives in the immediate area my children were quite close with them. After a few years I started to notice at 4:30 on the dot it was like a switch was flipped. He would speak to my Aunt in a harsh tone and be very belittling. So I would still bring the kids to visit but we would leave before he flipped that switch. When they would come to our house, I would make sure they knew we had plans at 4:00. Then the behavor became less predictable. My Aunt finally confronted me on it. We'd gone from seeing each other every day, eating dinner together several nights a week and at least one of the children spending the night there Friday and Saturday nights to only visiting for a few hours, to seeing rarely seeing each other. And then finally (currently) we only see each other when we both made the four hour drive to our home town for family functions. I told her that I was aware that my Uncle had many medical issues. I suspected at first he was "coming down" from the medications at 4:30 and suggested she speak to his doctor about it. She was greatly offended. I told her that 1) I did not want my children to see this behavior and think that it was acceptable. and 2) We love them very much but I did not want my children to see this behavor and let it change their opinion on him. Right now they think of him (still do 20 years later) as their funny Uncle that they love dearly that has become too busy to do the things we used to do.

I knew my Uncle's (and his families) mental health issues prior to allowing them to create that bond. Was I wrong? Maybe. They had a close, loving relationship for as long as it was good for them. When it became unhealthy I created logical reasons why we couldn't go to Aunties house, until it was no longer the norm.