Man, I would not worry about that at all.

You’re not hurting anyone (although that can be delicious in so many ways) and it’s a very common male fetish. Freud/Field Day: I think it so common because panties create the very slightest cover/barrier for the labia/vagina. I could be wrong. My fetish is fer nekkid bodies male/female/trans/whatever.

Don’t feel guilty! But don’t go through your neighbors dirty clothes, either.