Does he ever include you in activities he does with his daughter? And I don't want to sound paranoid, but are you very sure that he is single and not involved with anyone else? If you have any doubts, listen to your instincts.

Otherwise, I agree to talk to him about this. If he doesn't want to spend more time with you or is not able to, and that makes you unhappy, then you have a decision to make.

All the best....