It's not how often you drink, it's what happens when you do. If you don't like what happens when you drink, then don't. If you can't stand not drinking on those few occasions that you do, it might be more of a problem then you think. Just try quitting and see what happens. Keep us posted.