I'd say trust your intuition on this. If it doesn't feel right or you feel uncomfortable by his behaviour you have a right to leave. It might be that he finds you attractive but those are his feelings to work through and shouldn't impact on your work together. Maybe he is just being friendly and professional so you could give it longer but do trust yourself if something feels off.