You're right that in an apartment it's easy to hear all kinds of things; for months in the corner unit I occupy I heard a couple fighting violently on one side, and another couple having sex very emphatically on the other! If you're positive that it's not your workout they're hearing, then you could just carry on working out in your apartment, because they're going to hear noises no matter what. If you stop working out and they keep on hearing noises, then you'll have compromised your workout for nothing. It doesn't sound like the manager has actually told you to stop working out, and the manager may not have intended to communicate that to you. If he's only passing on a message that they've complained about noise then he may have simply been going through the motions of talking to you.

Perhaps ask the manager for clarification.