Could it be possible the real problem isn't physics, but math?

Physics is basically math written and expressed in a different way. If you struggled in math, chances are you might struggle in physics.