I have a dead pine tree, too. It will cost $1500 to cut it down!
Neither of you have to remove the tree. The tree is on your neighbor’s property, so it is his tree. Your fence is on his property, so it is an ‘encroachment’. No matter who put up the fence, the neighbor can sue you to remove the fence off his property. Sometimes, people can lose part of their land by allowing an encroachment.
You can get a copy of your survey of your property at the county appraiser’s office. Then you can show him the tree is on his property. But, I recommend you don’t even talk about it again because he’ll come after you over the fence.
That’s the way I understand it IME.