THE BOOK--Playing The Percentages In Baseball

I’ve done similar research in the past - only using actual production from older drafts instead of projected valuations.  As far as the meaningfulness of the data, of course, I ran into all of the same issues that have been mentioned at FG, of course.  It is mostly a process of giving a name and some “credit” to mostly anonymous scouts.  If you know the scouts well enough to drill down into their stories, though there are some neat nuggets.

I posted these results in the comment section if you’re interested:

I did the same kind of study several years ago measuring actual produciton from the 1987-1996 drafts. That helpfully gets away from the Brandon Wood is super valuable issue. It’s not a measure of potential value, but actual MLB production. I never published it, but the way I did it also takes care of some of the other criticisms that have come up. First, I calcualted a slot value for each draft slot. Back then I used BP’s WARP, but I’ve since changed over to rWAR. This is still in WARP. Then I determined a pre-FA slot value. Then found the pre-FA production for every player from those drafts. So for each player there was a surplus production above expected slot production.

Here are the 11 most productive area scouts from the 1987-1996 drafts.

Data is name, pre-FA slot WARP, actual pre-FA WARP, Actual – Slot WARP, some key players.

1. Tim Kelly; 6.0, 90.7, 84.7 WARP; Tim Salmon, Troy Percival, Chad Curtis
2. John Ramey; 10.9, 86.7, 75.8; Marcus Giles, Joe Mays, Jim Mecir
3. Joe Delucca; 22.1, 95.5, 73.4; Manny Ramirez, Charlses Nagy, Pete Harnisch
4. Guy Hansen; 11.0, 83.6, 72.6; Kevin Appier, Jeff Conine, Mike Magnante
5. Luke Wrenn; 26.8, 95.9, 69.1; Tino Martinez, Mike Hampton, Nomar Garciaparra
6. Roy Clark; 17.7, 86.5, 68.8; Kevin Millwood, Greg McMichael, Jerry Dipoto
7. Scott Trcka; 18.8, 81.4, 62.6; Scott Rolen, Jeff Jackson
8. Matt Sczezny; 20.3, 82.0, 61.7; John Valentin, Mo Vaughn
9. Marty Esposito; 32.3, 92.7, 60.4; KNoblauch, Mark Guthrie, Todd Walker
10. Ken Madeja; 4.5, 64.1, 59.6; Derek Lowe, Matt Mantei
11. Erwin Bryant; 2.1, 60.8, 58.7; Jeff Bagwell

I went to 11 so you can see signing one great player from a low slot (4th rd) is going to shoot a scout way up a list like this. In actuality all of these guys signed 2-3 good players tops.

And that’s over a 10 year period when some of these guys turned in a thousand reports or something like that.

My conclusion is that this is a great way to draw some attention to area scouts, but for all the reasons mentioned it’s just not a good way to rank scouts.

You need those preference lists for that.,

I’ve always been interested in asking someone this, and now that’s it’s come up I will.  Tango, do you have any insights into how teams actually grade their scouts?

Just for my own curiousity, I’d kill to have access to some old pref lists to compare to the draft slot values that I have derived.

Just like you say forcasters should release old forcasts to the public to further research in general, it would be a great boon to understanding the scouting process to have access to old pref lists that are completely useless to organizations.

I have no idea.  And if I did, I would be precluded from sharing that information.

I’d think Keith Law can help you here…

The best value pick in the 2009 draft appears to have been Mike Trout with LAA. Where did LAA have him on their pref list? If they had him high on their list, that would tell us that they did an outstanding job scouting the player and they deserve immense credit. But what if Florida, Houston, and LAA all had Chad James, Jio Mier, and Trout on their list, in that order. All three teams had it wrong; how much credit do the Angels get for having the last selection of those teams and stumbling into Trout?  In effect, they won because they had the losing lottery ticket, in your example above.

In any case, crediting the area scout with signing a player is really an outdated concept; when scouts used to travel the country, discover a player, and instantly sign him, it meant everything. Now, draft picks are really organizational decisions.  Any player selected in the first round will have been seen multiple times by the Scouting Director, National Crosschecker, Regional Crosschecker, and others, all of whom will have much more bearing on whether the player is selected or not than the original area scout.

Often times, the biggest contribution the area scout makes on an early round selection is knowing the player well enough to know how signable he is and at what salary figure, which helps clubs decide who they should select in a given spot.

All good points in #3.  The way teams credit area scouts with signings is definitely an anachronism.  You have to look at any high rd or high bonus books as team efforts with the area scout as the initial analyst, but far from the sole analyst.

When I looked at my larger sample of data I always looked for scouts that maintained success with different organizations (or at least different SDs and GMs) and had some mid and late rd successes as well. 

But you can never look at players signed and get beyond the small sample issues.

Even over a 10 year span, what you’ll see is a small number of scouts sign a few high rd players and have an expected productivity of say 20 WAR and a whole bunch who mostly sign mid rd or later players with single digit expected WAR.  And that is largely driven by geography - scouts in CA, TX, Fla, etc will have a greater number of high rd signings and a lot of other scouts will barely have any.

But any one player hits big and that scout will automatically look great.  That the example of the 11th ranked scout in my post above. Erwin Bryant scouted CT and NE and never signed any really high rd picks so his baseline is going to be very low.  One of those mid-rd picks turned into Bagwell so with that one player he grades out extremely highly.

nice post Tango.

In some respects, this is the same argument as WPA on a game-winning HR.

I’m pretty sure Gordon was universally thought of as being better than Zimmerman, Braun, and Longoria.

Gordon was supposed to be a stud.

Looking at the MLEs of Gordon’s college stats, he does seem an over draft at #2. Then that perception of greatness was confirmed by his breakout in Double-A the following year. Thing is, 2006 remains the outlier of his entire career, and now he’s seen as a bust compared to those expectations - but he’s been exceptionally consistent, putting up 340-350 wOBAs the past three years, that while not star level, is above average for a MLB third baseman.

The value of this kind of analysis would need to know the rankings. We can assume that every area scout loved Tulowitzki, Gordon, Braun .... anyone in the first couple rounds was by acclimation, no? So the value would be in the later rounds, where someone saw something and spoke up in the room, I think.