You're not alone and its a well known fetish. Since this doesn't seem to be interfering in your happiness, I don't think you need stop it. This is my opinion, but I think stifling a fetish, actually fuels it and makes it worse. There have been other posters here with this same fetish and there are varying degrees.

A fetish only becomes a problem if, you can't control it or its interfering with your inner peace and happiness. As long as you can carry on in a normal relationship, then you'll be fine. There are people who like to wear them - maybe find someone who doesn't mind fulfilling that for you. You're not hurting anyone. We had one man here who was very scared about posting this fetish and he felt much better when he got supportive replies.

http://en.wikipedia.org/wiki/Diaper_fetishism