That is up to you. Will going out with him help your anxieties or will you have a negative experience and reinforce your anxieties? Do you have a job or club that you attend? Do you have family or friends to spend time with? What about a support group?
Since I am a scientist at heart, I would write up a pros and cons list.
If I go out with him If I don't go out with him.
sdjkiduioj oasidfjioadfjio

This is how I make descisions when my mind is all spegettied up. It helps me to untangle the noodles of thought from each other. I'd also suggest that you try to think up some simple social activity whether it be talking to someone on the bus, talking to people on-line, sitting out on your porch where people can see you or whatever and try to do it daily.
As for sex drive, I don't know what that is. Could you define that for me? (Hehe. I don't have one nor have I ever had a sex drive.) Good luck.