The first thing you have to understand is that absolutely everything you do or experience alters your brain. Everything. However, it sounds to me like you have run across some sort of propaganda somewhere.

If porn is causing you distress, then it is a problem and you should do something about it. Otherwise, I wouldn't worry about it. Watching porn (for most people) is one of the most harmless things you can do.