Your certainly not crazy,

I'm sorry that your having to deal with a messy situation. Best thing to do is get all the financial issues and visitation things in place. You need to set boundaries for yourself when dealing with him.

He needs to be responsible for his son.

I hope you can get everything done quickly and you go about living your life with out him directly in it. Other than what decisions need to be made together for your child.

Good luck