Yes I have experience with this. If you love this location so much that it is the only place you ever want to go on vacation, then buy. But a second home can be a money pit if you don't use it enough.
Renting gives you the freedom & the cash to do other things besides pay a mortgage, insurance, maintenance etc. And to try other places too.

The people I know (besides me) who have done this, had in mind that they would eventually move to the other home permanently someday. Or use it as an income property. Lot's of choices to consider with a project like this.

Good Luck to you as you move forward