I agree with Trippin2.0, it's your (mutual) call. If you are okay with things as they are, that's fine - some people get along quite well that way, usually when neither party is interested in a label. If you aren't okay with his refusal to call himself your boyfriend or partner, that's valid too. It's a really tough decision.

Is he open to discussing this with you?