I think you may be jumping to a conclusion just a bit quick. I say always get the facts, all the facts. There seriously may be a good reason he hasn't responded that has nothing to do with you, perhaps a family issue, or out of cell range on a trip or who knows what.

When is the next time you see him? Discuss all of this with him then, exactly how you feel now. I'm sure he'll put your mind at ease about this toxicity thing, because I don't think that's true at all.