There is too much second guessing for me on what he thinks, feels, wants, etc. Only way to know is to ask him; not talking and just listening (two-edged sword, also means not communicating) and waiting for him to bring up what bothers you is manipulative to me -- he doesn't know that you want it brought up and attributing to him that he does know and is deliberately not bringing it up just adds imagined stuff to what's already going on in your head and cannot help the situation.

We know he feels like he is being made to lie by your not sleeping with him "in public". That is a lie of sorts if you are sleeping with him. It says you do not trust his judgment about his own sons or that you have a problem with he or other people knowing to what extent you are committed to him. It is your secret, he does not want the two of you's involvement kept a secret anymore and is disappointed that you do.

What could he have brought up in the conversation that you were quiet for? Why would he have brought it up? He has stated what he wants and how he feels, he has nothing else to say? He is probably not going to force/beg you to sleep with him in either event? When will you sleep with him when other people may know/find out? At some point, for the relationship to move ahead, it sounds like you will have to do that? You need to do a statement, "I will not sleep with you when others are around at this time and I will not leave my son alone when he is not at home. I would like to meet and get to know your sons better before we have a sleepover at your house and if both our families go away together, I would like to stay in a house or suite of rooms so you and I have a better chance of being intimate with more privacy." I think anything else is just stirring up imagined slights.

You know what he wants and he can want what he wants. Tell him what you will/will not do so he knows, uncategorically, what you want and will/will not do. There is no "argument" or discussion; you want what you want and that's fine. You will/will not do what you will/will not do and that's fine. He may/may not want to continue the relationship given what you will/will not do but there is no need to try to change what the other person wants, a want is a want, not a negotiating point. You may/may not want to continue the relationship given what he wants and does (he may be moving too fast for you and refuse to slow down, for example).