It's nice that students are comfortable around you but they're running you over. Some profs I'm close to, they joke around, while others, there's a risk of if you do that, you may get a hard slap and thrown out. The issue is, not only may it annoy you but what about students you like the subject? They want to learn it but other clowns may be goofing off. I'm in 2nd year university and since I'm taking courses I love, I hate whenever people around me are babbling during the lecture.

if this is after the lecture, then it's not too bad unless you're trying to answer questions.