You havn't been stupid here, this was definitely a comment to analyze.

I don't think it was so bad of him to say. It would make me feel a bit flattered and I would just let it go. Really, he might have thought he was just building you up.

I'm surprised his girlfriend doesn't try to keep you two apart and doesn't see you as a threat. I give him credit for staying faithful to her. If you push on this, you might end up losing the friendship.

I'd wait and see if he says something sexual again, then I'd talk it out with him and set things straight.