You need to open up with him and just be honest with him about what you just said to us. That you have desired him for a long time but want to wait for a little while to get to know each other 1st. He will be fine with that, I'm sure. maybe he was genuine when he said he just wanted you to lie next to him. He is saying he wants some intimacy, and because he knos you do, maybe that's his way of initiating it. If he knew you were wanting to wait I'm sure he wouldn't have asked you to come over. Please don't let your BPD ruin this for you. He sounds like a great guy.
At least you can even date. I haven't been on a date for 5 or 6 years; it's been so long I've forgotten count. I can't accept that someone could find me attractive, appealing or worthy. So at least you're dating. Now you just have to not let your head do itself in over little things which could be cleared up with open honest dialogue. Good luck. Keep us posted.