There obviously isn't a lot of chemistry between you two, so I would go with that. Just say you don't feel the chemistry is there and you don't think it's worth pursuing. Chemistry is important in a relationship and if it's not there now, it probably won't develop. I don't think it has anything to do with your sexual preferences. You might be focusing on that because you're not focused on him and you're searching for reasons why. Good luck!