Like I said, my biggest sin is the iced coffees from the supermarket which unfortunately don't come in decaf. But I've stopped drinking those now.
I still occasionally have a coffee at home but I add way more milk and less coffee now and so far that's been okay.

It's those times that I've had the high caffeine coffee and it didn't have any effect that are deceiving. It's tricky when you sometimes get issues and other times you're completely fine!