The issue of the phone strikes me a little different. I think people need to keep their noses out of other people's phones. That is not to say that his behavior with the phone is normal. It sounds sneaky, as does his behavior otherwise.

I don't think your distrust is based on your own anxiety issues, and don't let him try to sell you on that. The guy's just not trustworthy.