Saying "you can't control my friends" is a cold statement. When things seem cold and apathetic, it's time to leave.
It's not about whether the statement is true or not, it's about what's behind it - "stay away from me".
Even if it's not at all that way, it's somewhat that way, and it can definitely be better. You deserve better.