Fantasies are about you, like dreams are only when we're awake instead of sleeping. I would start noting them and then seeing if you can figure out what they are telling you (for example, that you are lonely or want companionship, are afraid of or don't know how to interact with others and form a relationship, etc.) and then work on that issue.

If you are able, I would start a friendship with the woman you work with, get to know her better and more critically so your fantasies might lessen as you come to know the real person, warts and all :-)