I don't think you should feel bad about it. I can relate to what you say when you describe fascination and such. My interest isn't spanking but women's panties. What you say matches my experience to a T. If I hear a lady even mention the word "panty" or talk about what types of panties they like to wear my ears perk up. Seeing a lady in panties is also extremely erotic to me .
Like you, I tried to understand why I was fascinated by it but came to no conclusion. But the way you describe your story is almost like I wrote it word for word except that the turn on or fascination is spanking for you and panties for me. Mine started around the age of 11 or so and has been with me ever since (26 year old now). Also, like you I am sort of picky about the fascination. When you describe that certain forms of spanking such as couples spanking each other isn't a turn on, I'm also picky in that only certain "styles" of panties are the most arousing to me (thongs and g strings). I spent many years distressed over it and feeling very ashamed that I was in to something that wasn't just "normal sex" or what you are usually fed by society. I just learned to accept that this was the way I was and not care what other people think. Just know that you are not alone there are many different fetishes/ fascinations including feet, panties, stockings, spanking, bdsm, etc and people are going through the same thing you are.

As a side note, if you went to a therapist they won't be able to tell you either. I've tried to figure out why with one and they couldn't tell me why. It just has to do with certain events that happen in childhood. For whatever reason, your brain paired spanking with a strong erotic response and it "stuck". I'm not sure if every person has a situation like that but it is not uncommon or something because lots of people have different fetishes. I imagine upwards of 10% of all people have a fetish in the sense of something that fascinates them that they acquired in childhood , they just don't talk about it. That's almost 750 million people so please don't stress about it