I don't think it's harsh. The inflection in your voice could matter when you say it. Maybe try for matter of fact? In a way that doesn't let your frustration with this person/situation trickle through?