What would I do? Likely nothing.

If your heart is telling you vegan is the way to go, then what anyone says about should be of little consequence to you conviction. You certainly don't have to defend it.