I know exactly how you feel. Fostering relationships doesn't come easy to me (I am an introvert) and even a toxic one was difficult for me to let go of. And it sometimes bothers me to this day that this person was unable to give me what I needed. But I was never more alone than when I was with her.

If this is a "friends with benefits" situation and you are not happy, then you have to do what is best for you. And that may mean telling him that you can be just platonic friends or bf/gf but not in limbo as you are now. You must be true to yourself. There is nothing selfish about that. That said, you should communicate to him in no uncertain terms that you need things to move to the next level. That it is a must for you. There is risk involved but the things most worth doing are often the hardest, as they say.