I don't think you're to blame. It's his need/obsession. I suppose some women would be okay with it, but I'm like you. I wouldn't feel comfortable about it.
I suggest you go to see a counselor. Maybe at some point he will go with you. But remember he is happy with the way things are.
I guess if you decide you can't accept things like they are--and he isn't willing to have regular sex with you--then you'll have to see about making other living arrangements.
I'm sad that this has become an issue. I suppose he was afraid to tell you before.