You faced that fear before and overcame it. If you did it once you can do it again.
If you had avoided him the fear would have been constant. You didn't avoid him and the fear disappeared.
He doesn't have to be your friend for you to lose your fear. Maybe the fear comes from something else.