Imho I'd go with your gut.
Yes to me this is fat shaming. Why only ask one person in a group a food question. Do not the other members count? Or their opinions? Wouldn't you want a recommendation for a place from several people instead of one?
If you saw this person become uncomfortable then he probably was. Next time you could jump in with your own food opinions perhaps & maybe ask others in the group too.

And I hate to say this but your mom's theory that a heavy person might have good opinions on different foods is also skewed. Don't you think?