I don't know what you are planning to study in grad school but if there are math prerequisites and you are having difficulties and being stressed now, before you even start grad school, then you may want to rethink what you want to study as I don't think the math requirements will get any easier?