Well, the funny thing about emotions and sex is that the emotions often sneak up on you when you least expect it. One of the four of you might be absolutely certain that he/she is "fine with it" one day, but over the ensuing weeks and/or months, find his/her feelings changing. Especially since there are more than two of you involved, I can see some potential for attachments and jealousy. And what happens if one of you suddenly gets turned off by the whole thing - how easy is it going to be for the rest of the group to handle the rejection? It's easy to say "well, it wasn't personal" in terms of what you all intended to do, but it's extremely personal in reality.

I'm not saying that it will happen guaranteed, but the potential is definitely there.