With or without BP II, a grinding situation can make for a lot of irritability too. For me it's a source probably as common as that stemming from the BP, so the situation's contribution is something to keep in mind. You're not alone
That your Pdoc said the other symptoms are hit and miss, that's kind of weird. When irritability is the main symptom,
a greater number of the symptoms must be present for dx. But the point is they still need to be met. DSM itself is very direct on that. Anyway, I just think that's weird of him.
I don't know if having irritability all the time is typical or not. It isn't for me outside episodes. When you mention the clothes, 2 things come to mind. One, being more sensitive in the senses is one of the signs I might be headed upwards. The feel, the sounds...some clothes are too noisy for instance. Two, it may be that you're an HSP (highly sensitive person). It was something my psych and I explored at one point. It's something you may want to check out.
Personal experience is having irritability in both depression and hypo/mania, but they seem to have a bit different quality to them. In depression it's more a frustrated overwhelm -- more personal, more like a (hate to say this) tantrum feeling -- tired, cranky, overwhelmed. In hypo/mania, it's more external. It's more likely to be lashing out at Other People. They piss me off with their slowness and stupidity. Just something I've noticed, typical or not I don't know.
Wishing you the best in sorting through all this. It takes time and your mood charting is a very good start in the journey, so keep up the good work.