You are NOT wrong. Not even a tiny bit. Keep those boundaries strong and do as you and your husband wish. Congrats on the baby!