If you're really uncomfortable about the relationship it may be a good idea to back off.

If you like it but feel very confused about your negative thoughts it may be a good idea to take some time out to resolve them.

Either way try to keep some peace of mind and sense of perspective. That is the most important thing.