I will make another suggestion that may work for you.
When you feel the emotions overwhelm you, instead of taking it out on the baby, go into your bedroom and scream into a pillow. This helps to release the emotions instead of suppressing them.
Or punch the pillow (with the intention of releasing built up overwhelm, not pretending it's your baby you're punching).

You seem to realize the baby's not the problem, which is good. Awareness is the first step to changing this behavior.

You certainly don't want to be in the position of watching your grown child taking his anger out on his children.
You can stop this cycle of abuse, and you are abusing your baby when you take your anger out on him. He is at your mercy, treat him with the kindness you wish you had been treated with. The dividends in the future will be huge. His well being is in your hands.