I agree, tell him you just want friendship. If possible do this with a friend nearby so they can hear and support you.

If you have a therapist get their support too, have them help you plan out how you can make sure to pace a possible relationship how you are comfortable, not based on the other persons desires.