I would say that no you are not wrong, and it makes sense that you do this. It's probably healthy. Unless you think it's wrong yourself. But it sounds like you are protecting yourself and setting / maintaining boundaries.