Just keep being nice and spending time with him. I've tamed many a feral cat and the key was just patience and time. There have only been two that were too wild and never tamed down so that can happen but they were never ever lovey dovey at all. The fact that your cat can be lovey is a good sign.

Please be gentle with yourself too, you're not a monster you just made a mistake.