Well, I think every man will find it difficult not to look at people they find attractive, but as you probably know that doesn't make them perverts. Are you insecure about his feelings towards you? That could be the reason you are so angry, sad and hurt. Your feelings are valid, even when you don't understand them, and they have a source.