Yeah, there's no problem here. You didn't cross the line. His girlfriend is being insecure. But does his joke about you ruining his relationship concern you? Maybe there's a mutual attraction and you know there's an element of truth to this?

The fact that this whole thing was initiated by you discussing contraception implies some level of comfort or intimacy. The fact that you're posting about it here seems to indicate it means at least a little something to you.