It really is hard to try and create something that really didn't exist. I have a similar situation with my father who left my mother when I was about 8 years old, and drifted in and out till I was and adult. Now, when we talk, it is a challenge to talk to him about things in life, about my marriage, about my life.
I can only say that it will take time to create that trust. Never assume that the trust cannot be built, but rather, be open to that possible opportunity. I am not saying to lay all your cards outs, but taking it one step at a time. Ensue a conversation about one particular topic or subject or event. After awhile, enter in another topic, so on and so forth, up until you feel that discomfort. That discomfort is just your defense mechanism letting you know you just need to take more time.