It's inappropriate to stay late on the job, after your work is done, or to show up at the workplace on your day off. Most employers have a rule against that.

Your coworker and others have been doing you a favor by telling you that your behavior is objectionable. If you really "look up to them," then why not take that advice to heart? What you are doing is trying to win an argument by trying to make words mean what you want them to mean. "Looking up" to anyone, firstly, means showing them real respect. What you're doing is not respectful, in the least. You are being "intrusive." That is a very dislikeable trait. Learn to curb that tendency. It's, basically, a form of attention-seeking. It represents an attitude that says, "Notice me, notice me, notice me."

You owe your coworker an apology . . . not an argument.