There's no "off-limits" with fantasy. My dad confessed to me (after discussing my own fantasies of Nazis) that he sometimes fantasizes about past girlfriends he had, and he's been with my mom for 25 years and they are both 100% faithful to each other. I wouldn't say it's inappropriate at all. Most couples DO fantasize about previous relationships or people they may know in real life. It's normal to do. Don't worry about it so much, or talk about it with your counselor.