There's a lot going on in this message. You say you're not a lady but you asked if you were being rude. Sounds polite to me.

I've been helped by just conversing here and I hope the same for you. The answer may be in your original post . . .
his family doesn't know you. Have they not known you for two years? You sound outgoing and as though you are exploring. So besides the "friendship" you having going with him, decide what you want for you.