Do you think he could make an exception? For example, maybe the group has 15 people in it. It wouldn't hurt him to have ONE extra person (in my opinion).

You could ask him for an exception. Tell him exactly how bad you feel, and tell him why you want him to make an exception