Your SO is totally crossing the line. Unless both parties have agreed beforehand that the relationship is non-monogamous, there's no reason for that kind of behaviour. Furthermore, mentally and emotionally "punishing" someone is a form of abusive behaviour.

You are definitely not crazy to feel that this is a problem. I rarely ever suggest this, but were I in your position, I'd give some very serious thought to ending that relationship immediately.