I agree with what everyone else posted. Your discomfort in his house is valid and it is 100% okay to not go back there. I wonder if you are getting some kind of intuitive sense that he is interested in more than friendship and that's part of why you don't want to go there?

It may help to practice saying in the mirror 'No, your place is too far, let's meet halfway at wherever' so you don't keep agreeing to go to his place.