Honestly, I myself have no problem dating a guy who smokes pot. Most of the guys I've dated, have done so. But maybe that's because I'm still in high school. I would say don't let what he does get in the way of how you feel for each other, if you really have that connection you will regret not staying with him.
Just don't let him get you into anything that you don't want to do. And if it makes you uncomfortable, tell him to just not smoke around you.
Hope that helps