I wouldn't worry so much about the heart. The heart beats faster/harder when you exercise, too. And that's not bad at all.

However, stress and anxiety affects your health in other ways, producing chemicals and toxins that aren't good for anything. Just remember that nobody has ever died from a panic attack. Thinking the opposite will just make it worse, so I'm really glad thats not the case!

Of course, if you already have heart problems then you might not want to strain it. You can always do some simple tests to ensure that it is in good health and not the source of your problems.