Without information on how specifically these people have affected your ability to follow your passions it's quite difficult to say what you can do about it. About the only takeaway I get from what you've said is that they've given you a bad reputation with complete strangers? First if they were complete strangers, they shouldn't matter. I'm not sure how these strangers are in some way an influence to your being able to fulfill your passions so I can't quite comment on that but.. I can say that if the people involved in bad mouthing you are in circles that are influential to your pursuits then, you need to find a new group to work with. Doubtful you can do much of anything to people that are intent on making things difficult for you. Question arises, do you even need to be in contact with these people that are having a bad effect on other people in your life?

You can change them, you can only change what you're doing. Change social circles and keep that private from people that cause trouble.

Don't let others, no matter who they are, influence whether you pursue your dreams and goals, ever.