Sure. I was the 20-year-old and "he" was the 45-year-old. It didn't work out romantically for us, but over 15 years later and he is still my best friend. I don't have "daddy issues"...we just really clicked well.
At 20 I never could have dated a guy my own age...the maturity just wasn't there. After going out with aforementioned guy for a year, I went out with someone 14 years older then me on and off (mostly on) for over 10 years. There were some issues due to our age difference (but it was mostly differences in personality...that was what really broke us up), and we did get along better when we were both older than when I was 21 and he was 35.
I really think as long as both people are honest and communicate (important in any relationship), I don't see the issue.