It's likely that they take you ignoring them as a weakness? I really don't know at this point, nor would I be able to give you a solid answer unless I knew more about the situation as a whole.

If the situation gets to the point to where they're harassing you, it would be best to file a police report against them and receive your friend's pepper spray and not be afraid to use it if needed.