You have a valid fear, statistically speaking (stats on violent crime against women do not lie here). You can overcome some of the fear by carrying mace with you (where legal) and maybe get a large dog to take with you on long walks out in the country/neighborhood or wherever you go that's canine friendly. This works for me. I don't necessarily fear men as much as I lack trust in them. I don't trust any men and that's okay. Don't try to force yourself 'over this' because someone thinks your issue is non PC or in 'poor taste' regarding the interests of gender relations. Whatever that's all about. It's real, you just have to stay alert for indications that it's time to leave or high tail it from a place or get away from some person, and otherwise live your life as normal.