I think that this is something that you really need to talk to him about. It isnt something that would be considered an "unnecessary fuss" but something that is something to be "genuinely worried" about. Sex is a key component to any intimate relationship. If it is something he is now withholding, it is important that you know the reasons why. He may be doing it because he feels he doesnt want to pressure you and the times that you do want it, you may not be clearly communicating that and signals are just getting crossed. There could be any number of reasons. But until somebody is brave enough to talk about it, its just going to keep stewing on the back burner with both of you possibly questioning what is going on.