I think you should not feel guilty for pursuing your own passion which is horses! They really can be quite therapeutic. Please stop feeling like you are the only one who should make sacrifices to keep this union happy and together.

You will learn to resent him if you don't clear this up soon. Time for a long talk until a compromise that makes both of you happy is reached. Big hug and good luck.