Why is it that I'm only attracted to women that will never find me attractive?

I don't think ny "standards" are too high. I know I'm not very attractive so obviously I'm never going to be able to date a super model or anything like that. But the only women that are attracted to me (and they are far and few between in the first place) I have no attraction to them.

Do I just need to completely give up on finding anyone I'm actually physically attracted to and just date someone that I can't stand looking at? Because apparently that's my only options...be alone, or date someone I don't find at all attractive.