You know you don't have to be super built to be healthy and have a fit appearance. Fitness is a great goal that we should all have but it should be something you enjoy that improves your life, not suck all the joy out of life. As long as you have adequate protein intake and a consistent workout routine, you should be gaining some muscle tone. You can troubleshoot these types of things to see were you may not be getting the results you want using reputable information. Building muscle boils down to physics, which works for everyone, including you.

On a slightly different note, is it that you do have a decent physique but you just don't consider your current gains "enough" (and be honest with yourself here).