Tell them they need to come over to your house, that you cannot be making these trips because of what it does to you. Then go about your business. I am sure they are capable of handling this situation once you begin to draw your boundaries with them. They will adjust. They will have no choice but to adjust.

I know this would be a big step for you. But IMO your mental health comes first.