Home > FNQB, NFL > FNQB: Examining how much Front Offices know about the Draft vs. How Much they Should

FNQB: Examining how much Front Offices know about the Draft vs. How Much they Should

When tackling a topic like this, I know I have to be careful to not wade into the deeper waters of psycho-analysis that are out of my level of expertise.  I’m going to be upfront here: this is a tough concept to evaluate.  There’s availability to common information, which all teams have, and my hypothesis (which will be supported below, but I make no attempt to “prove” anything) is that teams do not use enough freely available information when valuing draft prospects, instead relying strongly on historically flawed internal scouting.

There will be no Friday Night Quarterback next week.  Liveball Sports will be covering the NFL draft.

[picapp align=”none” wrap=”false” link=”term=ndamukong+suh&iid=7533119″ src=”b/a/d/2/Pacific_Life_Holiday_7538.jpg?adImageId=12507602&imageId=7533119″ width=”500″ height=”333″ /]

First, I have to set up a few parameters.  It has been suggested by reputable sources that the NFL Draft is a crapshoot, not dissimilar to a horse race.  To an extent, this is a good analogy, but the data doesn’t hold up to suggest that teams know nothing about the players they are drafting.  Courtesy of research done at Pro Football Reference, and using their AV metric, we can see that the players that are drafted higher/earn the higher grades end up having the best careers.

You can look at that list and see that the difference between the first overall pick and the 6-10th overall picks is about 12-15 AV, the difference between the first overall pick and the end of the first round is about 30 AV, and the difference between the first overall pick and the end of the second round is about 40 AV, or about 2/3 of the total value of the average first overall pick.  From that point on, the difference in AV from the average 65th overall pick to the 97th overall pick (3rd round) is the same as the average difference from the back end of the third round until the end of the draft.

You can take from that: draft positioning matters.  This of course, is not new research, it’s something that has been proven in the past.  It cannot be summarized that the NFL Draft is a crapshoot no matter how much anecdotal evidence can be offered.  Rather, it would be more accurate to say that there is very high variance in draft choice quality.  In the last fifteen years, both Peyton Manning and JaMarcus Russell have been first round draft choices.  Neither is remotely close to the mean AV value for the first overall pick: Russell’s truncated career has produced roughly 12% of the expected career value of the first overall pick, while Manning is a hair over 250% of the expected value for the same pick.  Still, knowing nothing else, the team that holds the first overall pick should not be expecting to draft a player that reaches the heights of Manning, nor falls to the depths of Russell.  The team that uses the first overall pick on a quarterback should EXPECT to get something like Carson Palmer on the polished end, or Michael Vick on the toolzy end.

If you ignore the developmental upside of draft prospects (and no one does this, it’s purely a hypothetical), risk-neutral metrics suggest that you could only draft a player that is more valuable than a Palmer or a Vick (neither of which is a consensus top ten player at his position) through either good fortune or a market inefficiency.  I think that really drives home the point of the NFL Draft: either the averages are depressed by poor decisions, or the averages are depressed by the expected value of high-risk moves which backfire more often than not, but teams are willing to take these risks because great teams are built around a few great players, as opposed to trying to just beat the average pick every time.  In all honesty, it’s probably a mix of both.  There are calculated risks, there are bad decisions, and both contribute to sub-optimal draft outcomes.

I’m proposing that there’s evidence that the risks teams take may be unnecessary: that you can avoid high variance players early in the draft without necessarily avoiding players that might some day be great.  The further down you get in the draft, the less available that these “universal” prospects are, but that’s okay because teams that pick at the back end of the first round and beyond don’t have to spend the mega bucks to take the calculated risks that teams in the top half of the first round do.  The penalty for flat missing on a player at pick no. 24 isn’t much, so it might make some sense (even putting aside the ability to develop players) to take a raw athlete with considerable upside, but for a team picking at no. 9, it’s a poor allocation of assets to make the same pick.

Essentially, the crux of my “teams at the top of the draft don’t maximize” argument is that:

  • these teams aren’t selective enough when they are really the only teams that are in position to be, and
  • teams make the mistake of assuming that the prospects they know most about have depressed upside compared to prospects they know less about, in other words: teams mistake the absence of information for upside

The second point is the biggest point.  Incomplete information is not anything except incomplete information.  While this statement sounds intuitive and indisputable, you would be shocked how much more productive any team could draft if they treated information they do not have as information they do not have.  Furthermore, I believe I can teach anyone to predict the fortunes of future draft prospects fairly accurately, simply by comprehending a statement that sounds rhetorical.

This gets back to my initial point: the over-reliance on internally-generated scouting reports vs. the freely available information.  Completeness.  An internal report is not an internal report if the person responsible for the report doesn’t come to a bottom line conclusion on the player.  At that point, the decision-makers will evaluate the player on film, and compare their findings to the internal report, all while putting together the big board.  What happens a lot, particularly with underclassman, is that based on a few games (for guys that have only played a year or two), you’re either going to get a positive or negative internal report, followed by a positive or negative grade from the people who will be in the war room.  The higher picks are going to be the ones who get the positive-positive reviews.  Those are the players who are liked by “the organization”, with limited internal dissent.

Limited.  Internal.  Dissent.  This is the common thread with ALL picks who eventually bust.  Positive-positive grades imply that a first-round prospect is a can’t miss.  Sometimes, though, there will be suggestions in the freely available record.  Want proof?  Let’s look at NYJ LB Vernon Gholston’s (2 years, 11 tackles, 0 sacks) college career.

What Scouts See

2006 OSU 47 21 26 0 8 0 0 1 8 0
2007 OSU 37 25 12 0 14 0 0 0 0 0

SACKS = boldface, SOLO TACKLES = Underlined

The Bigger Picture

2005 – no production

2006 OSU 47 21 26 0 8 0 0 1 8 0
2007 OSU 37 25 12 0 14 0 0 0 0 0

2008 – no production (left school for NFL draft)

Notice what happened here when we used freely available knowledge to fill in the full picture.  Without in anyway diminishing Gholston’s accomplishments at Ohio State, expanding the relevant sample just raises a lot of questions about his viability as a top ten pick.  You can say that in 2005, the reason that Gholston had no production is that he was buried on the depth chart behind better players, and you can say that had he come back in 2008 for his senior year at OSU, he would have been even more dominant than he was as a junior; you might actually be right.  You would unquestionably be missing the point:

  1. Gholston was the 6th overall pick, a point in the draft where teams should be being super-selective in who they should be taking
  2. You’re assuming that you’d be right because your conclusion based off two years of film evidence that Gholston is an elite pass rusher.  In other words, rather than leaving the freely available evidence as the evidence, you’ve now extrapolated into the unknown based on a confirmation bias
  3. There is no reason to actually do this.  Gholston is either worth the 6th overall pick based on what he showed in college, or he isn’t.  The projection of “what is” is being skewed by “what could/should have been.”

You want an easy, efficient way to screw up your draft projections?  Start ranking players based on a mix of what they did and did not do at the collegiate level, instead of just on what they did do.  Take Georgia Tech WR Demaryius Thomas for example.  What he did in college: for his two most recent years, Thomas used the advantage of playing in Paul Johnson triple option scheme to run three different vertical routes very, very well, turning plays that the athletic quarterback strings out into long gains.  Prior to his junior year, he was a 15-16 yard per catch guy averaging about 3 catches a game, and 3 TDs a year.  As a junior in 2009, he added an extra catch per game to his average as the GT offense added another dimension, he got in the end zone 8 times on just a handful more catches, and he shot his yards per catch average up to 25.0 per.  He will not play in 2010, having committed his efforts to the NFL in that year.  That’s the complete pre-draft picture on Demaryius Thomas, you tell me how much that’s worth.

This is not suggest that Demaryius Thomas is/isn’t worth a first round pick.  It’s just to point out that if history is any guide, the team that likely picks him will be doing so on a scouting expectation for his ability to adapt his skill set to their offense.  They might be right or they might be wrong.  They might be overdrafting him or they might be getting a steal.  All of this is missing the point, which is that you can just grade Thomas based on what you actually have tape on: the way he ran a few deep routes against one on one coverage.  That ability has a tangible value to an NFL team.  If that’s worth a first round pick to add a dimension to a pro passing game, then a first round grade is justifiable.  The assumptions that internal grading will make about the kind of thing that Thomas can/can’t do is what is going to separate his draft value from his actual value.

Furthermore, calling Ndamukong Suh a “sack master” probably doesn’t follow from the statistical evidence on his career (one 8+ sack season in four years as a starter), however, there’s more than enough there and on tape to show that he’s a dominant force on the interior, and well worth a top five pick.  Again, the key is to not assume anything, and just let the evidence reflect the evidence: the more Suh played, the better he got.  He’s not the best defensive athlete in the draft, and he doesn’t have to be.  The evidence suggests that he may not be an 8-10 sack DT in the NFL.  It does not suggest that he’s isn’t a guy who can fit in anyone’s scheme, be a force on the interior, and make plays far more often than he misses them.

In my estimation, Ndamukong Suh’s 2009 season was not any more or less dominant than was Demaryius Thomas’, or was Jimmy Clausen’s, but there’s a reason that Suh grades out as an elite talent in this draft, Clausen grades out as a nice player who could help a bunch of teams and wouldn’t help a bunch of others, and Thomas is a guy that just does a few things well, and probably only helps a small handful of teams.  Will front offices take them in that order?  Chances are that they will, but also that they are willing to inflate the value of all of these players on their big boards because of what they’ve done most recently.  What they’ve accomplished recently shouldn’t necessarily matter that much more than what they’ve shown they can do at other points, but it’s certainly projected differently.

If you don’t assume that Jimmy Clausen would have a spectacular 2010 season at ND, or you don’t assume that Sam Bradford would have a necessarily healthy 2010 season at Oklahoma, it’s hard to project either as a top five pick.  Again, there’s nothing to suggest that they aren’t capable of replicating their most impressive season to date, but that’s exactly the point: assumptions = bad.  Just don’t include them in your draft projections.  If you need to fill in the blanks to make general sense of a guy’s career in your head, just use a basic, run of the mill, regression to the mean expectation.  Generic numbers can substitute for lack of information for perspective reasons.  If they look completely out of place with the rest of a guys’ career, well, good.  If you add an 8 game season of slightly above average production to Sam Bradford career’s figures, you get something resembling Rex Grossman’s career.  Then you would apply the film evidence of Bradford’s supreme accuracy, and poise in the pocket, and the synthesis of everything gives you an accurate projection.  If that’s a first overall pick to you, then he’s worth it.

Due to an aversion to the freely available stream of information, and a wrongheaded assumption that their internal evaluation methods know best, I believe that the average value of a high draft pick gets depressed by an artificially high bust rate.  Teams should not be required to project the best players in the first round and the worst players before anyone plays a down (this requires them to be more accurate than self-interest requires), but on the contrary, the difference between good prospects and bad prospects should seemingly be more apparent than it has proven to be.  The efficient, self-interested team should be highly selective when dealing with a big money slot, and more accepting of the fact that a questionable college prospect is not necessarily the one that has the most impressive upside.  In most cases, the best prospect is the one with the best resume.  The complication of this fact is — I believe — the cause of the great variance in the quality of the players in the top 15 picks.  In an efficient NFL draft, the highest variance players would be selected later than they currently are.

Categories: FNQB, NFL Tags: ,
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: