I think you may be transferring these feelings onto yourself, then transferring them back onto the other person. This probably has something to do with your anxiety you speak of.