Yeah, this is a great question! It’s really hard to tell, even if you’ve known a compulsive liar for years.
We’re under the impression my dad really does believe the lies he tells everyone, though it’s really hard to understand the behaviour, especially when lying has had worse consequences than the truth would have - at best, it’s just rather bizarre.
If it is an act, there may be ‘tells’ if you know what to look for, and are fortunate enough to be in the right place/time to catch them, but I’m really just speculating, here.
I’ve also been told I have said things I either know for a fact I didn’t, or *might* have said but had no real recollection of saying, so found myself racking my brain trying to figure out if I really did or not. Generally it’s just an awful feeling, being put on the defensive like that for who knows what reason. I’m sorry you experienced that.