I can understand the anxiety with labels too. My best friend had that when we were together, and he used to panic when I mentioned the future or even that I loved him. But years later, we both have no labels, still very much in each other's lives and happier than ever. He is quite comfortable knowing we will spend of our lives together, and you know what? It's good enough for me.
Do you really need the label? As hvert said is he open to talking to you about this? I see nothing wrong with it, as long as you make each other happy. If that's okay with you too, then there should be no worries.
It ultimately comes down with what YOU want out of this relationship. If you are insecure of other women or it just being a casual thing, then yeah, I would talk to him about it. But if things are great and you enjoy each other, do you really need the label?
Good luck with whatever you decide, we are here for you