Your body builds muscle and strength, what was once hard becomes easier, you don't have to expend as much energy doing it, like Dh is talking about. So it is important to change it up from time to time. Add something new, do things in a different order. I wouldn't worry tho about waiting to add the swimming, cause there is always something else you can add or change to get past the plateau.

If you notice your exercise to be seasonal than it's easy to change it up. Swimming indoors in the winter + walking + your restorative yoga. More biking in spring + jogging + hiking + swimming.. whatever you want it to look like. Different activities like this use certain groups of muscles more and in a certain way, so good to be well rounded. Also maybe tennis or something like that might be fun for you in the warmer months. Having a wide variety also helps one to keep from getting bored with the routine. You live in a great area for all this stuff too!!

Plus you have had a lot of stress, I don't feel very hungry when I am stressed.