I would tackle the teacher.

Make it clear you aren't happy about your son missing recess for those reasons. If you aren't happy with the response, see the principal.

At my child's school the teachers jump through hoops for you once you've gone above their head