Just tell him you are not interested, I used to work in sales, we have thick skins and don't take rejection personally. Rejection is usually 90% of the job