I think your annoyance is justified.

It sounds unsafe to go and therefore grossly inconsiderate to insist. But if they are going to insist anyways, it will be instructive to see whether your boyfriend prioritizes you and sticks up for you, or prioritizes them and leans on you.

Therefore I don’t think you should try to walk on eggshells with him, so as to avoid annoying him “again”. In my view you should state your feelings and intentions in a civil manner. It will be valuable to find out, and reflect on, what his priorities are in this situation.