Wow, creepy creepy. This guy was definitely up to something. My radar was up just reading your post! I would definitely call your property management company, tell them this guy's behavior was inappropriate, and that under no circumstances do you ever want him in your home again. I would also notify your local police. Yes there is nothing they can do legally but you want this on record. Make sure you keep records of your own of who you talked to, when, what time, what was said, etc, etc. Also go back in your memory and write down clearly what happened, in sequence, the day he came to your home. Include your feelings - scared, frightened, etc. Make it as complete and detailed as possible.
In the future, keep your screen door, if you have one, and your door locked, and front light on. I don't want to scare you but if this guy has marked you as a "victim" he may try to come back even if he's been told not to, under some false pretense ("I left my xyz tool in your home") If you see him within 100 yards of your home, don't even hesitate - call 911 immediately. When you go in and out of your home and get in your car, lock it immediately.
I don't say all this to scare you - that is not my intent. Chances are good that he'll never come back. But it pays to be safe, to take that extra time and precaution. If something like this ever happens again, DON"T HESITATE to tell the person "You need to leave now or I'm going to call the police" and then do it. Don't be polite and continue listening. You're very, very lucky that guy left when he did without hurting you. The way you said he was rambling it could have gone either way, and you need to be strong enough to protect yourself. ***** being polite or nervous - your life could be at stake!
By the way, look back over what you said in your post. You said he did "nothing aggressive". Yet you also said at one point he was "holding a large wrench in his hand" and at another time "he moved toward you twice". Those are both aggressive actions on his part.
Please be safe MDD - please post again and let me know how you're doing - I'm concerned about you