I'm feeling a little better after I found this article: New Research Shows Rat 'Lingerie' Really Gets Rodents In The Mood | The Huffington Post
Basically it was an experiment done on rats where they learned to associate more arousal when their partner was wearing a certain type of "lingerie" (weird I know). I think a similar thing happened to me, somehow I conditioned myself to like my "fetish" , probably subconsciously ,and the fact that there is a biological explanation for it makes me feel better about the whole thing.