Well it would bother me also.

Is it possible to sit him down and probably again have a heart to heart discussion about how you do need to have time for yourself ?? I’d really have to push push and push the issue until he understands I need and deserve to have “ me time”

Maybe telling him it would be nice if he took her out for lunch if he was insistent upon seeing her?? You can do your thing in peace.

I feel for you