The issue really isn't the incestual thoughts. The issue is that it's bothering you having them. It's not abnormal to fantasise about all sorts of things. I've had fantasies involving family members before too

If it makes you 'happy', and it's not harming anybody, then you have nothing to feel bad about