Hi there. I hope i can be of of some help.
Firstly, I do understand the being "ashamed" of buying a vice...I smoke with my roommate, roughly a pack a week, and I wish I had a quarter for every time I've had a cashier stare at me like some subhuman species for asking for a pack of Marlboro Reds. -_- I wouldn't advise drowning away your sorrows, as that's not a healthy response (especially if alcoholism runs in your family), but I don't think you should feel ashamed of buying it, per se. You're an adult, and you have the ability to make those decisions.

lol So...I'm not advocating you buying a drink, and I am hoping you don't, but I'm hoping you wouldn't feel bad about it if you do...does that make any sense?
But speaking more to the point, I don't think your relationship is going to improve as long as he does in fact still have a wife.

Some part of him is always going to feel something towards her. He did, after all, marry her, whether it was a "fling" or not. Even if he doesn't love her, the notion that he is A) supposed to and B) the social ramifications of the whole thing are going to bog him down, and thus affect you. I don't think he is going to be willing to change that any time soon.
That being said, and I hope I don't offend in this, as you know your situation better than I, but I think you'd be doing yourself a service by leaving him and finding someone else. I don't mean this to offend or judge you by any means, but someone who has shown a willingness to have "other women" in his life isn't the one for you, you know? Why would he show you any more courtesy than what he has shown others? I think you can do better than this "man."
Additionally, I don't think you're a homewrecker by any means. Please, don't think of yourself like that.

I think her and him both did a rather admirable job of wrecking their own home...I'm not sure there was much left to wreck by the time you got there, based on how you describe it.

You're not at fault there.
You can do much better than this boy Aquarius, and I hope your next post will be you proving that statement right.
My absolute best to you, and you're in my prayers.
Hugs,
Harley