Don't feel guilty. I don't think losing him would be all that bad, given what you wrote above. Some time living apart may help you figure out what you really want out of life and whether or not you want him to be a part of it -- that may be what he's really worried about, that you are going to realize that he's a dud and drop him.