Its just stress and hes mind. Even if he says he doesn't worry about it while you have sex, hes self consciousness probably is. See if he works to much and is stressing, maybe there is something on hes mind that he keeps thinking, maybe its even hes erection he is thinking.
I doubt he doesn't fancy you anymore.

I had issue with my erection to, and once i spent a week with just my friends and having fun all the time, i didn't have time to have stress or be worried about anything and i had my strongest erection in years