Just text him and tell him. You're going to hurt his feelings whether you reply or don't. If it were the other way around, wouldn't you rather he just tell you that he doesn't see the chemistry rather than you waiting by the phone? He's probably wondering what he said or did wrong.