If this is happening 4 times, you may want to look at how you are communicating with these guys. Are you flirting with them, possibly leading them on, without being aware of it? If that is the case, it is time to establish boundaries and only talk to them as a friend, flirting shouldn't happen if you are in a committed relationship (unless your partner is ok with that kind of thing)

That being said, if you already have these boundaries up and engage in no flirting, then you have just had the bad luck of running into people who want to be more than friends. You can't be faulted for that.