The things that would bother me most are him not following through and always being late (this might be just my thing).
But consistency is essential for therapy progress- I see he can be pretty consistent with meds so why can't he be the same with proposed/agreed treatment?
You like him, he had help you so I would talk to him if I were you- skip the porn thing if you really can't bring yourself to raise it but give him the list.
He could change to some degree accommodating your needs- I think a good T should be able to do that for his clients.
So yes, I would stick with him providing you two at least discuss the cons and he tries to do stg about them.