In general~ I have no problems saying no but in this case, i believe would set up a few minutes to stop in. Just to meet his kid and say hello. Only because it sounds like there's been some chit chat and he's trying to get his life together and is proud of his motivation.

If someone told me that i needed to give 100 percent of my attention to my child~ it wouldn't go over well. I can think of a hundred reasons why but for one it implies that I can't have a personal life when i have my child. That sounds lonely for a person trying to get their life together. I might joke around with it and say "sorry dude~ I'm done with kiddy swing."

I am not saying I'd become friends but I would take the blame for not wanting to form a relationship rather than put it on him and telling him how to spend his time with his child.

It's a tough situation~ I wouldn't want to be in your shoes but I wish you the best.