I believe that the relationships you had in your childhood are an indication or directs your expectations of relationships when you're older.
In your case: you knew your grandparents loved you, but they didn't show it that much (if I make the right conclusion ofcourse), so you're kinda used to it, you don't know anything else.
It's a good thing that in a relationship you turn "me" in a "we", but don't make it a "him". I think you kinda lose yourself in a relationship and when it's over, you realise you're still your own person, with or without him.

I advise you: if he doesn't make you happy, treat you like you deserve to be treated and know you can do better (maybe you don't know it but I do), then go. Walk away from it.

I don't think you're expecting too much. It's like, in this relationship you're in now, you make it about him, and he makes it about him, so that's two people that make it about him, which is great for him cuz he's Mister Important.
A relationship should be about giving and receiving (now I sound like Joey in Friends, lol), and it's a two way street. Not: him receiving and you giving, no, he gives and gets, you give and get.
Now you should GIVE him the finger (or whatever) and GET outta there
If that is what you want and if he's not making you happy, cuz someone out there can make you happy and can make you feel worth it and loved and will make an effort for you.

And about the observation, I don't wanna compare him to anything but...If someone likes to observe monkeys, they're not going to the zoo everyday, right?