I definitely think anxiety is very different from mania, but maybe not because of the reasons you cite.

Firstly, at least my anxiety is often irrational, it's generalized and sometimes-to-often does not seem to have a cause.

I often don't feel fear of any one thing when I'm anxious.

Although my anxiety is generalized it can get worse or better (fluctuate) through the course of the day, depending on what does and doesn't trigger it.

I personally think that people can confuse anxiety with hypomania (not really mania) because it can make you very fidgety and full of energy and this can be mistaken for the extra energy of hypomania. It can also cause insomnia. I think of it as confusing too much caffeine with hypomania, similar symptoms but entirely different.

The only other thing I'd add, is that when I get hypo/manic my anxiety melts away, which is one reason why I miss so much being hypo.