Do the interview. If you would get the job, you would just work on the same floor, you wouldn't work with him or anything, so it shouldn't be a problem.
You could talk to him about this, but you don't need his permission.
Good luck!