I don't think the occasional low dose is harmful, but long term use (longer than 4~6 weeks) can lead to addiction, anxiety, depression, sleep disturbances, memory loss, and eventually dementia.
When I was in drug rehab I roomed with 2 ladies addicted to benzos and they were in much worse shape than the heroin addicts.
Personally, with my history of drug/alcohol abuse I wouldn't touch ANY benzo with a ten foot pole.