That's usually been the case with me, too. However, with my husband, I'd say I held on for at least a year, maybe more (it's been 7 years now). I am not sure whether it's intimacy in my case - but I usually do gain some weight as I get more 'comfortable' in the relationship, consequently making me more 'uncomfortable' sexually.
Vacations or at least getting out of town once in a while helps immensely.