Remember Rick Ankiel? At 20 years old, he started Game 1 of the NLDS against the Braves in 2000.
He took a lead into the third, and then had a complete meltdown. He threw five wild pitches, walked four and that was about all she wrote for his whole career. Not only did he not have what it took to pitch in the playoffs, but his performance so shattered his confidence that he never succeeded again in the majors.
And conversely, look at Mariano Rivera and flash back to 1995. (I know... he plays for the bad guys... I'm just trying to make a point here.) Rivera was a starter when he first came up... made 10 starts actually... and finished the year with a 5.51 ERA. Nothing special by far. But all it took was 5 1/3 innings of shutout relief in the first round of the playoffs against Seattle to see that this was a guy who thrived when the game was on the line in a clutch situation.
Rivera had "it" and Ankiel didn't... and there was nothing up until those key series for each player that could have predicted their success. There wasn't a scouting report out there that could have told you enough about their mental makeup to clue you into whether they would wilt or shine in a tough spot.
And even if there was, its still a matter of what happens on the field. A lot of people show their character in different ways. How about John Rocker? Sure, he was mean, angry...etc... things you'd probably normally want in a closer... but he turned out to be a complete head case.
So, while velocity and control might be key measures of effectiveness to a scout, a lot of the times what is really predictive of success has nothing to do with a person's natural talent. That sometimes makes predicting success an exercise in character judgement more so than it does a job of looking at someone's historical track record. Should scouts be conducting character reference interviews? I wonder if they do at all.
What about as you are building a company? Particularly in venture, when you might be asking people to do things that no one has ever done before, a lot of times, you find yourself betting on character. This is made so much more important because of the size of the staffs you are adding to. If employee #5 is a bad hire, its a lot worse than a bad hire for #5000.
So what are the keys to checking someone's character out and also the rightness of fit with an organization? Certainly, you've got to sit them down and put them in front of as many people you know and trust as possible. That was certainly a key for me when I got hired. While Brad and Fred never bothered to check out my resume, because they saw my work first hand in my due diligence of their fund, they focused in on my references and the impressions I made on people in my office. I'm quite sure that my interview with Kerri, because I was going to be sitting right next to her, was probably just as important as whatever I had put on my resume, had they seen it.
I guess its a little bit like dating. You just don't know exactly if you're going to be a match, and no profile, quiz, etc. is going to prove compatability for you... sometimes you just have to take a chance on someone you get a good feeling from... someone you wouldn't mind facing a little bit of the unknown with.