I too have had this fantasy and it has troubled me as well. I have finally come to the conclusion that it is not a problem as long as I don't act on it. It makes me feel better to know that this is common. I was unaware of that.