thickntired,
I think it is difficult to remember most things at age three, other than snippets and "gut feelings." That could explain why "your head" has trouble remembering your brother. I mentioned remembering snippets of school (like a favorite dress or growing a lima bean plant in a milk carton) but I do not remember the house I lived in from zero to 7 years of age. Memory is an amazing thing!
My personal feeling is - because you mentioned you still get messed up looking at his pictures - that your "heart" remembers your brother. Sometimes little hearts can also feel survivor's guilt. And unless little hearts are allowed to grieve and work out the loss, they lock it up. Some will throw away the key, but the memory is still there, maybe manifesting itself as pain stored in the body. I say this from personal experience only.