I think the thing that I've learned about relationships is that time is on your side. Not everything needs to be decided in the here and now. And stepping back to clear your head is necessary at times.

If you feel it isn't going anywhere with him, then it's probably not going anywhere. If he's not your best friend by this point, then what's the point?

How long have you been together?