Did you get the sense he was really fine with it or were you picking up on some ambivalence from him? Perhaps he said "fine" but he was actually ambivalent or even wishing you wouldn't go there. And maybe you were able to intuit that even if it was somewhat unconscious on his part because it seems like you're pretty attuned to him (maybe even more attuned to him than he is to you).

Regardless of his intent though he did actually say it was fine, so you should go ahead and visit the office building (guilt free) if you want to!