"lover" often (usually?) means someone you have sex with/make love to. Not per se someone you love. Your ex is right assuming it's more than a friendship as "lover" implies you two have sex.

If your bf said "my love" or "I love you" it would be something very different.