If you dont really love him then you shouldnt be with him.

Everyone deserves to be loved and to have love in their life. Be with the person who you really want to be with. Dont stay with him because of religion or any one elses reasons.
Follow your heart.

Caution though. The guy you really like might turn out to let you down when you come back to him. Just be aware, you might lose them both.