It's definitely a bit disconcerting, but perhaps not in the way you mean. I'm not so worried about failing to achieve my dreams because I didn't get lucky.

I'm worried I'll fail and never know if I failed because I was unlucky, or because I just wasn't good enough. That's what makes me uncomfortable.

I get over it pretty quickly, but it's still unsettling, if only for a moment.