Well... if you're really worried... just don't go. Tell your partner you had second thoughts & decided it was something you just didn't feel comfortable doing. I agree with SadGirl, though. I think you could just stop by, take a quick look around, report back to your partner, & let him take it from there.

I would recommend not showering & changing at his place. And don't allow yourself to be dragged into the situation any further. From that standpoint, this becomes a matter of personal boundaries. Here are links to 3 articles, from PsychCentral's archives, on the subject of personal boundaries in relationships:

Why Healthy Relationships Always Have Boundaries & How to Set Boundaries in Yours

3 Ways to Develop and Maintain Healthy Relationship Boundaries

3 Steps to Setting Healthy Relationship Boundaries