From what you say, you don't have a reason to say no...and I think it might be a good for you to accept the fact that he likes you...

Make sure it's a daytime, public "hang out" though...it's safer and there is sure to be something about your surroundings that you can make a topic of conversation if there's an awkward pause.

Have fun!