It really depends on the relationship you have with that person. I have some friends that we are close enough that we do this all the time. On the other hand, if it was someone who didn't have that relationship with me, yes, it would be rude.