I have had that happen, but at the beginning, like when H suggests sex.

What has helped for me is to mentally put the kids to bed or send them out to play or sit them down with a good book and assure them that the adults will be doing adult stuff and they will be protected. Still working on that, but it is getting better. H hates it when a 5 year old shows up in bed. He says it makes him feel icky. I am grateful that he understands.