Are you crossing hierarchy or departmental lines? Some people will socialize just within their level or department. Crossing hierarchy lines can be problematic in either direction. At work, it can look like favoritism.