Thank you for your advice. I bumped into him at a party. I said hello, smiled and that was it, I ignored him otherwise.
The next night we bumped into each other again and we went on mostly being on our own but we exchanged a few words. It was distant but a bit friendlier.
I did what you suggested and watched the guy. He is a flirter (4 women glued at his every word) but I noticed he was more intensely flirting with one girl he told me was "great" the night he met me.

So I figured I should find out more.
We ended up 4 people after the party. He brought each one to the metro station (clearly one of them thought he was going to have sex with her, I could tell and I'm pretty sure he already had sex with the other one, another "friend with benefits" of his probably).
I was left (and I saw the next thing coming). He invited me over to his nearby place for a drink and quickly came to speak of sex but in really crude terms that made it clear he wanted sex without emotions. He talked about great dirty sex he had with a friend. I am not a prude at all but the way he talked about this was so detached, cold and almost dirty.

I let him speak. He said that after the first night he was interested in me but that he felt kidnapped at the party the next evening, given the feeling he couldn't talk to anybody else (ridiculous, I never said anything like that, I just found it odd how he ignored me in the beginning and then seemed so defensive and half-hearted, asking me private things and then jumping up in the middle of it to go back inside). He made me feel totally confused and the only thing he came up with was "we're not married", "you make us sound like an old couple"... Clearly he blames me for ambushing him, for causing a nightmare. He said when I left without saying bye, it was actually a relief and that I was overanalyzing everything.

He added that I killed every interest he had in me on the first date and that now it was casual. He said that he was talking to another girl and wanted to be free in case it worked out (this is within a week of meeting me). He said she was a friend but he was hoping for more. That if it didn't work out, maybe he would consider me :-o
He was really rubbing it in that he wanted to sleep with other women ("so you can't say, I didn't warn you")
He said that it was normal to have open relationships, dating several people (with lots of sex of course) and that love could develop or not, but that there was absolutely no guarantee.
He said he didn't trust me since the argument. When I asked why, he got angry: "here we go again, analyzing everything!"
He mentioned "F*** buddies" lamely saying that he didn't mean that (yeah he did).
I told him that it was very convenient...for him. That it just meant that someone would be hurt and that someone would be me. He said that "if only one person loves, it's called a disease".

He made me feel like a needy, possessive creep, that had to be kept at bay.
He didn't seem to care one bit about how hurtful he was being. All he wanted was being clear.

So I told him that actually when he contacted me "to have fun", I got his message 100% (which is true). I thought it was a one night stand. He mentioned we were on a " first date" that evening, not I. He invited me to the second evening, not I, nor did I expect it.
He told me that we were the same, soul mates. I didn't really answer (too early).
All his compliments were otherwise on the sex and my body.

When I told him there would be no casual sex, he said "I thought so" but when he heard that I initially went into it as a one night stand (which he refuses to call like that: "let's not label everything"), he got excited and friendly and thought that there was a chance for sex. Again I said no.

So in short: he got interested in a girl to the point of thinking she's a soul mate, then loses interest after one argument, not caring a bit about her point of view, distorting it to a farce, then declares it's casual and yeah maybe it could develop into love but in a casual environment and she should really consider it.
He said that's how his ex-relationship with his big love developed (yes but that one ended, didn't it).

I have a hard time believing he was interested in the first place.

When I refused the sex, he immediately avoided any physical contact, even innocent and he became emotionally cold and withdrawn.
He let me check my email (I did that on purpose to check his reaction) and sure enough he checked if I wasn't trying to go into his mailbox. Remember, I am a controlling freak, who will suffocate him!
When I left he refused to talk to me so I just said I was lame to blame me for destroying the chance of a relationship and to say "I don't trust you" and letting me figure out what it meant on my own (I guess he doesn't trust me not to smother him). He said "yeah you're right, do it on your own. Keep the stress. I don't want it."

I am so angry because he made me feel like I destroyed everything because I seemed possessive and irrational at the second "date". He is a serial dater, obsessively into sex, but he plays the romantic soul who saw so much of himself in me and who wanted something serious but I ended up being not trustworthy.