What makes you think he's mad at you? He smiles at you when he sees you and he messaged you. To me that sounds like he's afraid of getting rejected again. Stop responding to him for now. Once would have been enough. He may be confused. First it's "Go away! Go away!" and not it's "Come here! Come here!"

Take it easy with yourself and with him. Good luck.