You're just not ready to move in with him yet, I think. Just tell him you need some more time, he should understand.
That's not bad. Not because you moved in with person A "soon" makes it logical to move in with person B early too. (Not that I think you moved too soon or fast or whatever, I just didn't know any other words for it, lol).