Hello, guyinlahore, and welcome to Psych Central! I am so sorry your brother and his friend took advantage of you that way, especially at such a vulnerable age.

And, sad to say, these experiences do tend to come to haunt us and affect our adult sex lives. As you say, too, they led you to become passive.

I really encourage you to start seeing a therapist. Sometimes issues do reappear if they haven't been resolved earlier. It would be good to work through the pain and the feelings.

Also, you might try posting in the Survivors of Abuse forum here. Many of the folks there have also been sexually abused.

I hope you will be able to start working through these memories soon. Your life will be so much better when you do.