Only you can really know what happened to you.

If in fact you were abused in time the memories will start to flood back and then you will have to deal with it.

Not everyone remembers stuff that happened to them, a long time ago they blocked out their memories, i've even done that even though i was never really abused as a child but when I was 16 and up.

It's hard to say hon, if you were or not. If you have memories of yourself being afraid of him and what not then anything is possible, you said he use to hit you with his belt? You were abused by him? Maybe that is the fear? The snap of the belt reminds you of the whipping you got from him... does that make sense?