If your minister is counselling you, he is subject to the same code of ethics as any therapist, and that code says very clearly...*NEVER* get sexually involved with a client. Also, extra-marital affairs are certainly not acceptable for clergy.

None of us can tell what is in his head, whether he may think that he is "in love" with you. But, he has crossed several ethical boundaries with you, which potentially has harmed your own recovery. And he has taken advantage of your trust. I think wisewoman's suggestions for what to do are good ones. You need to find an ethical counsellor who can help you through this.