Grandfather or even great grandfather works; I got a picture of my great grandfather doing my genealogy work and at first I thought it was my middle brother the eyes looked so alike!

My husband is 6' 2" and his ex-wife is 6' 4" but their 3 sons are all taller, the "shortest" is 6'6" and there's no one in their past that was that tall, though their mother's father was over 6'; my husband's father was only about 5' 7" and I have a brother-in-law who is that size too as well as another who was around 6' 6" (my husband's mother was 5' 11" or something). So you really can't tell where the height/burliness might come from.