I use xanax in manner similar to what you've described -- that is, infrequently, as needed for stress; maybe once or twice a month, tops, and for years without any issue at all. It seems to be the most troublesome (taking more than should be, etc.) for people who drink, which I don't, and there are some other combinations that can be dangerous as well. But taken properly, it's a very safe drug, which is why it ultimately replaced valium in the prescription toolbox.

Personally, a whole milligram would knock me out .. my prescription is also for .5's, and I've actually ended up breaking those into quarters (they split really easily), and sometimes only need one quarter, which I still find really relaxing -- so that would be my only suggestion, to try sometime taking less than you are (unless you've already done so more than once on your current medication set), and see if it achieves the desired effect in a way that is still helpful.

Definitely check in with your doc about how you're using it -- I'm sure they would be more than happy to know how you are using it responsibly