This is most likely anxiety. When you are having that casual sex that does not even excite you internally, you have nothing to lose and nothing at stake - you do not care what the lady will think of you and thus you are anxiety-free. Being free from anxiety lets you last longer.

Another thing is that when are internally excited, you ejaculate faster, which makes sense - you are excited.

Have you tried non-penis-centric sex with some of the women you care for?