There is something seriously OFF about this! And i personally think you should be a little concerned.
The fact that you are posting on here suggests to me that you already are concerned.
If it was his child then obviously it would be different, but she is not and this child does not need some guy coming in and out of her life who used her mother and then left her.

I am not buying the whole missing her daughter thing i have feeling that the OW is also included in this scenario, im am not saying that he means to do anything, but i know children are beautiful and they create quite an impresssion on our lives but she is apart of her mother if that makes sence so in away missing the child may in a sense be a crouch for missing the mother as well.

However, he may have no affection for the OW and just miss the daughter, but it is not fair on the little girl in this scenario and your partner should just stay away from her as it will just confuse her. If he truey cares for this child then it is best for him not to be in her life, as it will just cause conflict in yours and his relationship and send the child mixed messages, and he will probably just end up dropping out of her life again and she will only suffer like she did before when he left.
He needs to think long and hard about his choices, he already broke this little girls heart once, and probably the mothers and YOURS as well and enough is enough.

Some times we think were happy and we believe what the other person is saying, but some times they LIE.

Just look after yourself.