It seems to me that if you are husband and wife, that makes the two of you a family. You two should be the 'first' family priority. The extended family (your parents, his parents) are second to the relationship between you and your spouse.

I agree with what you said, that it is important to support your spouse in front of your extended family. It's a little strange for him to bring this subject up anyway, especially if you are not fighting with his family. Do you know why he is doing that?