I'd be annoyed out of context, because it makes it sound like you can't handle anything. Fragile is a little condescending.

In context (from what you said in your last post) I would probably be concerned that the frequency and severity of my episodes was increasing. I'd want to know why that was happening for sure. If they are going to make a statement like that, they better have some answers/solutions for me.

It's a very odd way of putting it...