Kind of a similar situation. I moved 3,000 miles away from my hometown to start a new life. Not long afterward, my pregnant daughter followed me out here. After the baby was born--so did her father. She's now expecting her second. I guess he has as much right to be near his grandchildren as I do, but I hate the fact that he and I are living in the same city again. He'd come running back to me in a hot minute if I'd allow it, but I won't. We tried reconciliation a few times, and it never worked. Well, recently he tried to connect with me on Facebook. Not only did I reject him and block him, but this repulsed me so much that my hubby messaged him, "Do not try to contact my wife again. Thank you." You know what? Ex answered him back, "Do not threaten me or I'll call the police, and I'll contact (where you work) and get you in trouble." Well, we knew the police would just laugh, so we weren't worried about that. But hubby is a bus driver for the city. It would be easy for ex to pose as a disgruntled passenger, and make up stories claiming hubby did something. So we alerted management that my ex had made this threat, and told them his name. Now they know, if they get a passenger complaint and it has my ex's name on it, it's bogus.

You aren't being petty. What your ex is doing, is called gaslighting. He's trying to make you think that what went wrong between you was your fault, that you are the one who is defective.