I'm not sure if devastated is the right word, but OK.
I just don't want to loose it. I don't want to push it either. Nevermind what particular feeling it is. It's not a bad thing and I guess he'd understand if I wanted to express it.
I don't want to interfere with his family life.