It would bother me too, but I do think it is a natural way that men sit when they are relaxed. It might actually be a good thing if interpreted that way. He's not upset by anything you have to say.

If you feel comfortable about it, I would try to bring up your discomfort to him. It is a lot of openess. It would bug me.