I agree with Skeezyks that this attraction to men dressed as women is only a problem if it's affecting your life adversely. Does your wife know that you are having sex with men? How does she feel about it? And do you personally feel any shame or are you comfortable with yourself? I think those are the most important questions to answer.

Also, just wanted to add that I hope your sex has been safe.