Or, do both- go visit, but make it clear that your time is valuable, too. An "I feel" statement would be helpful, such as "I feel like you don't respect my time when you schedule over our plans."