I worked at some great big schools with tons of faculty. Of course, there were people who only wanted applicants who had been at Harvard their entire life. But they were the minority. I really think you would be surprised how many decision-makers had to struggle to get where they are. My dad was the first person ever in our family to go to college. He became a rather famous and powerful MD, through hard work and perseverence. But his biggest break came when the world-renowned chairman of a tremendous training program heard my dad's story (not at all like yours, but a very impressive tale of pulling himself up by his bootstraps against all odds nonetheless) and picked him over a bunch of people from the 'fancy' schools. That break led him to what he became. That chairman picked my dad over all those Harvard and Stanford people. He chose the person, not his background.

Just something to keep in mind. Just because someone selects applicants for a living doesn't mean they have led a gilded life. They may have been homeless.