I would recommend that you have a serious conversation with this guy and make clear what you are wanting from him and visa versa. Do you want to be friends? dating? relationship? or FWB? Correct me if I am wrong but you mentioned that he wasn't openly affectionate with you before you both had sex together? Then afterwards he was more affectionate and didn't pull away from you when you touched him?

Could he be just warming you up for another sexual encounter (FWB)? I would make your boundaries known to him so that he doesn't make a fool out of you or hurt your feelings if you are wanting to be more than just FWB.