Manipulation is a pretty broad concept.

Children manipulate all the time, and their parents manipulate them. At work you are constantly being manipulated, and manipulating other people. It's unrealistic to think that it would never happen between partners, even though in a perfect world, everything between adults in a loving relationship should be communicated openly.

Manipulation is not the same as expressing feelings. Even if those feelings are negative, frustration, grief etc.

But you're right, if he wants to spend time with you, he should just ask.