Many people have a hard time with certain types of people. For me, trying to talk to someone who doesn't let me finish my thought really irritates me, and I clam up. Or, if I just don't have anything in common with that person. One thing that you may consider, many people who seem very confidant tend to also be outspoken, outgoing, etc. Could it be that you just don't feel as if they want to talk to you? Many women are much better listeners, which could explain why you can talk to them much easier (wish I had that problem ).
Just my 2 cents, for what it's worth.