Maybe he abused his wife and maybe that's why she left. Maybe kids observed him treating her bad. You said he didn't always treat you well so I bet he didn't treat his wife well either. Or maybe he treated them bad. Maybe his substance abuse was too much for them to handle. Maybe they see therapists and were advised to keep communication to a minimum.

My dad isn't bad but he can be difficult (he hasn't been past two years as my mom battling cancer), but when he was difficult my t advised me to keep communication to a minimum to protect my own sanity.

My mom also says that dad was always great (warm and attentive father.). That's baloney. Neither me nor my brother think that. My dad is ok now but he was NOT warm and attentive when we were growing up. He was not emotionally available etc and was very tough to deal with. Not what mom saw. So I wouldn't go by what their mom thinks.

And he probably never complains because he feels guilty

It's still sad and difficult for you. But try to take thoughts of it as like others said you don't know everything. You love him and it isn't always provide objectivity. Stop worrying about them. Hang in there