Aw Rainbowzz ...
This has got to be frustrating (and thanks for the heads-up about what *I* should expect). How about pumping during the times when he refuses to eat, and then freeze the milk? That way, you won't have to worry about your supply being impacted, and then you'll have some extra milk on hand for when you're ready to wean him from breastfeeding. Or, if you don't use it, you can donate it to your local milk bank - there are tons of women who would really appreciate it!
I would/will feel just as conflicted as you, so I understand.

hang in there