KB, BM has great practical suggestions.

Under your circumstances, you are not in a way greedy. Rather, you are being deprived of private time with your lover. If the guy is out of the house irregularly for short perios of time, then what you and the woman do basically amounts to stealing some private moments together. That is not good for either of you, but especially for you, KB.