You wrote "gambling, drinking, and occasional drug use". I do not know what you meant by occasional drug use (what drugs? how frequently? etc.), but to me, gambling alone would rule this person out. I also do not know what you meant by drinking but if it is heavy drinking, then, again, to me it would rule this person out. Bipolar is OK, but gambling and heavy drinking are not. You have only one life to live. Plus, I agree with Anika that he is toxic for you. I know from experience that when YOU start to, quoting you, "lose my family, friends" then it is sheer abuse and you need to drop the guy.

I sincerely hope that once you drop him, you will eventually be able to regain your family and friends.