If you have gas money to spend; I would find someone local to you who is occasionally willing to drive you somewhere and stay/pick you up a few hours later? I would work on figuring out my own solution to the transportation problem you feel you have. That way you could be independent.

Can you start a transportation "kitty" to save up just for the purpose of being able to occasionally visit that town?

Do you have any place in the town you don't know well/would be "new" and easier to be in? Could you find a place to stay occasionally that wouldn't trigger you as badly? Then you could come/go to it as you wished (local cab or friend) instead of being dependent on the others.

I would think about occasionally entertaining them at your town; inviting them for dinner or something at a restaurant (or your place if that's possible). That way I wouldn't feel as beholden to them for inviting me and/or turning them down so often and you wouldn't invite them to stay or anything so it would be time-limited.