How much stress are you under being it's your first year of college? It is possible that is contributing some to how you feel, I know college can cause a lot of stress and especially if you already have any mental issues that can interfere with mental functioning, concentration and memory.

It is possible the cannabis use could have contributed, it does not effect everyone quite the same so hard to say. I personally have smoked it for like 5 years and have not had any effects like that from it, if anything it makes me feel more clear headed and able to sort out my thoughts. Either way if it did contribute avoiding it should get rid of any detrimental effects it may have had, not really sure it would do any permanent sort of 'damage' as far as studies go cannabis does not damage brain cells, however not everyone reacts well to it.

Also you shouldn't smoke on account of other people, if you aren't enjoying it and feel its detrimental it would be best to be honest with your friends, any who would stop being your friend just because you don't want to smoke aren't real friends to begin with...however I kinda know the feeling of indulging in substances when I didn't particularly want to just for the social interaction(not a good idea, but I have been there...not with cannabis though as I do truly enjoy it).