I differentiate between transference (feelings about a person from our past who can be identified being put onto someone now) and projection (our own stuff being put onto someone now, not necessarily related to a particular someone in the past, could be something we do that we don't want to admit to)

I don't know if that would help you, deciding if the guy reminds you of your Uncle Leroy when you were 5 or reminds you of you in a way you don't like?

If you are fighting for "control," for example, when you're with him, you might not like not being in control and what's under that in you (perhaps fear or feelings of inadequacy), or you might remember when one particular baby sitter use to boss you around and you thought you were old enough and resented not being "in charge" yourself and he reminds you of the stupid baby sitter and, dang it all, you don't need a baby sitter anymore! :-)

I guess another way I'd put it is whether you're fighting a memory of some sort (transference) or a straw man (projection). I had to work through transference with my T relating to my stepmother but sometimes realizing it's just the stupid babysitter in this one instance and not an ongoing problem with "all" situations like this one (transference) is enough and I can easily light a match to the straw man myself.