I don't think it is trivial at all.

If you really don't want to go out with him, it would be best that you set the boundary immediately by saying that you want nothing more than friendship.

If you would like to go out with him, and the idea of a date frightens you, just think of it as coffee with a friend.

I have been out of the dating game for years and years so I am not really sure what to say. I hope what I suggested isn't lame.

Hugs