Hot pepper is added to suet to deter squirrels. I've also added hot sauce to the feeder itself because squirrels don't like touching it. So I inferred that the hot pepper needed to come into contact with the rodent. I didn't consider that the smell might offend them.

It might be helpful to think of what stuff might be attracting the mice if said stuff can be easily removed or put into air tight containers.