I find this a little confusing actually.

There's kind of an unspoken thing among guys that, to quote Jack Nicholson, you don't "... rub another man's rhubarb." So dating exes of friends can get really awkward. It usually has nothing to do with the girl, but more about not wanting upset their buddy.

This doesn't necessarily sound like that though. You might talk to him a little more about why it haunts him and makes him feel bad. Is it because he doesn't want to hurt your ex or is it that he maybe feels threatened? That might help narrow down the problem.