if he is saying he is falling in love with you then you telling him you would like to spend more time with him should not scare him at all. just be open and honest with him about it.