Would couples counseling be an option for you? It might be useful to have a neutral 3rd party.

I don't enjoy being around high people myself, having grown up in an alcoholic home. I can understand this being pretty disturbing for you. I hope you are able to work something out.