Ps, I read my post and I don't mean he's taking advantage of you as in hurting you- I mean like calling you up for lunch and not explaining to you his coworker would be there and you show up feeling like a mess because you weren't expecting the other company. I think he took advantage of your wanting to be with him.
I hope you didn't take anything I wrote the wrong way. It was all meant to be supportive but it may not have come out that way and I apologize if that's how it's interpreted. But I think you may be giving him too much of yourself for "free". Let him work for it.
Is that too much attitude?