Maybe you should indulge him, act out the visualizations w/ him. Except of course it would not be w/ your old "friend w/ benefits". Talk to him before you consider doing this.
Both sexes have a sense of jealousy to some extent, especially when it comes to sex and previous partners.
Although you shouldn't have to sacrifice anything you're not comfortable with. Just an Idea.