I personally believe kids are highly influenced by the actions of their parents when they are of a young age and while it can still happen at any age it tends to has less of a pull to it from the negative influence being seen when they are older.
I would be more concerned if your husband drank (got drunk) around your son when he was younger.