While I personally have no problem with marijuana, I do have a problem with people not respecting the ethics of others - especially when they are in a relationship. While his offer to include you seems to be a concession on his part, if you strongly feel it is inappropriate he should take heed of that. Frankly, it sounds like peer pressure. Sit him down and express that to him.

If you were to insert any other word for putting your foot down about marijuana use being inappropriate in the relationship (ie. porn, drinking, gambling, abuse) this would be a deal breaker I'm sure.