It isn't for me to decide what's appropriate for you but I'd be wondering why a man with a wife and two kids wants you to come over and hang out with him. Also too... the dating talk between you and this man seems a little odd. Seems like you would be curious about these things too.
Or am I missing something here?