No, it's no fun granted Laura; if I could reverse the clock on this, and run a thousand miles away from it, I would. There's no point in pretending that it's an optimal situation, except to say that it is what it is, and we just have to try to make the best of it.
But as for your manager, don't let him exploit your situation, as it sounds like that is what he is doing - in essence, playing head games with you. Tell yourself that the problem doesn't lie with you, but really it's all to do with him. So he's the loser here. IMO.