I don't think anyone can tell whether or not it is an issue of which you should be concerning yourself. I think it could be at some point, but there again it's not necessarily going to be either.
I tend to lean towards what Innocenjoy said, the bipolar could be wrong. That is always a possibility, but without seeing a professional in this area, you'll never know. I can understand where you're coming from with not wanting to go there, but perhaps you shouldn't rule it out completely if this is concerning you. Not all doctors are crap, not all of them are going to try to force medication on you and send you on your merry way. There are some who will make the effort in helping you get to the bottom of things.
No one can say if this will become problematic because everyone is different. For some it will be, for others it won't.