I've always lived in large suburbs (100,000+) or large cities (1 million+), and a couple of years ago moved to a town of 13,000 pretty much in the middle of nowhere. 98% of it consists of dairy farms -- there's not a lot of "town" to the town, LOL.
Before that, I moved from a city I'd lived in for 10 years, had plenty of friends, etc, to one I'd never been to and where I didn't know a soul. So yeah, I have a bit of experience with this.
It takes time to adjust. You are kidding yourself if you think "a few weeks" will get you over the culture shock of the move you're considering. It took me a solid year to get comfortable in the little town where I am now. It took me a solid 2 years to stop crying on the flight back to my new big city after moving from the one I was comfortable and settled in. It's all doable, but it will take you some time.
I have a question for you, though, as it's something I was about to go post about myself, and it seems to fit.

My T is always asking why I'm always looking down the road instead of in the here and now. I get her point, which is that a heckuva lot can happen in a given timeframe -- particularly in the 3-4 years you're talking. Heck, right now I don't know where I'm going to be next MONTH, much less in 3 years. I know where I'd LIKE to be, but life has a way of screwing up the best-laid plans.

It's great that you have dreams and goals, we all need them -- but maybe break your focus down into "in 3 months, I'll do this, in 6 months I'll do this" just in case you get thrown a curveball with your health, finances, loved ones, etc. And then remind me I said that when I'm trying to come up with my next 5-year plan.
Good luck.
Candy