Campervanman, I have similar experience. It seems like people are happy to see me and it even seems they enjoy my company a lot, but NEVER ever call me. If I do call someone, they always have "previous engagement" and "will call me other day" (but never do), but what hurts the most is relationship with someone who thinks he is giving you his best by letting you see him once a week.
I don't believe for a second it's "something you say" or something I say that is making them back away. I am not sure what it is. Sometimes I think it's because I look cold and distant but most of those people know me very well, and they know it just looks like that since I am shy and introvert. So, that's not it.
Yes, my secluded island with a cat and my own tropical garden is also waiting for me, I hope, some time, somewhere.