Adding to Leed's great advice, here is how i would answer your question.

It's time to give up only when you've done all you can to fix things. In other words, do the counseling thing as Leed said, the sign it might be time to quit would only come after you've tried this or if you found out that he was unwilling to attempt learning how to communicate and work together on things. I mean that's the only sign I can think of right now that would say to give up at this point - if he just won't budge about it, he doesn't really want to make it work. But you haven't alluded to that idea in your post, so I won't assume that's the case. Bring that up to him and let him know you want it to work, that you love him, but this part of the relationship needs to change and make sure you don't place the entire blame on him but that it's something you both need to work on and you'd like him to work with you on that. Go from there.

hope this works out! *hugs*