If I were that uncomfortable with the situation, I'd take my son and move back into my own place.

The adults can continue dating and reconsider living together after the children are grown and have places of their own.

That may be the best solution for all involved.