Hello Melaney: No, your way of thinking is not wrong at all. These are simply the conflicting emotions we all have. It's just part of being human! I'm an older person who's been married for over 30 years!

I love my spouse.

But much of the time, I simply want to be left alone.

I have this recurring fantasy of going to live in some spiritual community where all I would do all day is to quietly meditate... alone.

I'm not going to do it. But there's certainly a part of me that would like to. I think there's a part of most of us that wishes our lives had turned out differently.
By the way, I hope you are having an opportunity to get out of the house & enjoy yourself occasionally... even if you do need to tell hubby where it is you're going. Everyone needs a break now-&-then...