I allowed myself to become socially isolated, and as you know, to come back from it is so difficult. I can relate to you not wanting to become isolated again. As long as you feel you have proper boundaries I feel it is right to keep whatever friendships you want. Opposite sex friendships don't always lead to sexual interest. That seems to be a rather big generality. However, I also think keeping to some guidelines like not seeing your friend alone is probably a good idea. Perhaps you can explain to your h how lonely you were when you were socially isolated and you don't want to go back there. Also, could your h be asking (maybe subconsciously) for a little more attention?