THE BOOK--Playing The Percentages In Baseball

**I can bet you that there’s some A players that are better than others, and that you’d really want to move some of them to A+.**

This is a pretty silly statement.  Based on that logic, we shouldn’t give grades at all, because I can bet you that there are some B+ players that are better than other B+ players, and some B players that are better than other B players, and some…

Get the point?

I interview applicants for my alma mater.  We are asked to rated applicants in various categories from 1 to 5, one being outstanding and five poor.  We are allowed to add a + or - to each of the ratings.  I give out ones sparingly.  To me, a one is someone who is standard deviations above the mean.  A 1+ is someone who is so out there that I may not see him or her in a lifetime of interviews.  (I met an academic 1+ last year.  He had done graduate level research, published papers and some of his work was in the process of being patented.)

The A+ is there for the player that comes along some day who is so outstanding that he can’t miss.  I suspect the A+s never actually get rated, because they make the majors so quickly (think Ken Griffey, Jr.).  Someday, when John rates a player A+, everyone is going to stand up and take notice.

My attempts at nitpicking.

Gretzky: Not nearly physical enough. Will have trouble staying effective on offense once the pros start jostling him, and no one could be good enough offensively to have a dedicated set of enforcers whose only role is to protect him, right?

Magic: Doesn’t shoot well enough to be a pure wing player, but doesn’t play good enough perimeter defense to cover quicker wings, let alone the opposing point guard, so it would take some guts to let him play the point on a full time basis. Furthermore, if he can harness his unique offensive skills and become a point guard, it’s probably too much of an injury/fatigue risk to have him play defense in the post. Ideally, he’d be in a zone, but the NBA doesn’t allow zones. There’s real danger that, for all his offensive gifts, Johnson will be a defender without a position, severely cutting into his value. It’s probably best if he simply concentrates on playing power forward.

Dave/1: serves me right for posting on my way to the office.

What I was trying to say is that if you intentionally remove one level of category by making it so far above the next one, you are then going to cause an issue for the next level.

Basically, the gap between B+ to A- and from A- to A and from A to A+ should follow some sort of distribution, be it uniform or normal or what have you.  By making the gap from A to A+ such an enormous gap that it effectively will be a one in a million shot, then all those who should be A+ are going to be clumped into A.

Do you understand more my reasoning?

To put it in more simple terms: because no one can be considered in Babe Ruth’s class, we won’t have any Hall of Famers, and so, we’re going to instead lump Ted Williams and Willie Mays with Paul Molitor and Andre Dawson.

Sickels’ reasoning is more like when the judges don’t want to give a 10, because then they get boxed in.  But, after doing this for 20 years or however long John’s been doing this, he’s got enough of a handle to be able to create a new class of players. 

And, if by perchance someone comes along that is even more polished than Strasburg and his A+, then he can always give that guy an A++.  You have to be able to distinguish someone who tools-wise, out of the gate, is an equal to Felix/Verlander/JJ, and someone who is still a step behind them.

Where did John have, I dunno, Strasburg, Pineda, Latos, Bumgarner, Kershaw?  Were they all “A”?  Or did he have Strasburg as A and the next 3 as A-?  (And where did Kershaw fit in?) If he was able to separate Strasburg from the rest, then fine.  But then, did he end up putting several others with those 3?

John gives a “+1” to this comment:
“John is grading on a scale that tops out at A.”
-
So instead there being some lumping at A+, there is a larger amount of lumping at A that acknowledges there really is no such thing as a “can’t miss”.
That’s my take anyway.

David/2 -

You implicitly point out the need for normalization when different people are doing grading. I may grade more harshly, or leniently, than you.

When I was in software management in a large company, we tried to grade employees (ultimately, in theory, so that pay could be largely in line with performance), and we even attempted to make fairly concrete definitions of what those grades should represent. Yet we still met among ourselves to try to normalize people’s grades.

Tangotiger is suggesting that by making A+ unattainable, Sickels is compressing his distribution at the high end, and is thus providing less information than he otherwise might. One possible solution might be a shift to a numerical grading system - maybe he’d be more willing to give out a 97 or 98 than an A+ - but that risks suggesting greater precision than his grading might actually have.

The other thing is that John may be following some sort of distribution, but it is simply scaled down one notch.

That if he used A+ as a classification, that simply means that everyone who was an A gets moved to A+, and everyone who was an A- gets moved to A, etc.

Basically, the net effect is no change.

The best way to see what is happening is to see the counts for each classification, and see if the distribution is smooth or not.

Tango/8

I looked at Sickels ratings a couple years ago.  Here is the data from 2003-2009 (2003 is the first book that he self-published).

These are yearly averages:

A: 7 (0.7%)
A-: 16 (1.6%)
B+: 54 (5.3%)
B: 85 (8.4%)
B-: 92 (9.1%)
C+: 240 (23.8%)
C: 514

The C group is actually truncated.  He does at least 35 prospects for every team and (I’m pretty sure) gets down to C for every team, but some teams have a lot more than others.

I think from a numbers perspective, it seems like he probably should have A+ ratings to separate out the Strasbergs and ARods from the next 6 guys who will usually get As.

He’s probably a bit too flat with the Bs and B-s too. 

But I do like how his grades compare to basic top 100s.  The way it works is that he’s saying there are 7 elite prospects, then a next tier down to ~23, then a next tier down to ~77, then another tier down to ~160.

As someone whose looked at a ton of top 100 lists over the years that breakdown intuitively works for me.  I think it also jibes with Victor Wang’s prospect valuation work for THT several years ago.  There’s a break between elite prospects and very good.  Then there’s a mushy middle where it’s hard to differentiate (25-75) and then a bigger group extending out of the top 100 that are basically the same.

Let’s see.  From C+ or better, we have 6 grade classifications.

If let’s say we hold constant that we want 240 players at C+ and we want 494 players, what kind of distribution should we expect?

Well, if you look at the right-tail, and start at 0.5 SD from the mean and continue to the right in steps of 0.5 SD, we get this distribution:
240
147
71
26
8
2

That’s 494 players, with 240 matching John’s.  John clearly clumps too much at the B+ to B- levels, if we want to follow a normal distribution.

Best way to think about how your distribution should come back is that once you are in the right tail, each group should be a certain rate lower than the next, and at a subsequently bigger gap.

For example, in my illustration above, the 2nd level is 61% of the 1st level.  The third level is 48% of the 2nd.  And so on.  You get this:
61%
48%
38%
29%
23%

John is doing this:
38%
92%
64%
30%
44%

So, his distribution is not smooth, and the result if you get clumps.

Here’s one alternative distribution, if you start right at the mean point of 0, and go in steps of 0.5 SD to the right:
189 (240 C+)
148 (92 B-)
91 (85 B)
44 (54 B+)
16 (16 A-)
5 (7 A)
1 (0 A+)

That’s 494 players, but using 7 scales instead of 6.  I put John’s numbers in parens.

We see that John’s biggest problem is that he separates out the C+ from B- far too easily.  In a normal distribution, you have 337 players for C+ and B-, and John has 332, which is a fantastic match.

But the dropoff between C+ and B- is simply too steep in John’s list.

He pretty much nails everything else!  That’s a pretty fantastic distribution that he does frankly.

He would do well to give out one A+ for every 500 or so C+ or better prospects.

But, other than the C+ / B- separation, John’s distribution gets an… heh, A- from me.

Well, Sickels is entitled to use whatever rating scale he likes, but I might ease the standard for A+ a little bit. Maybe one or two discernible flaws might be okay, as long as everything else far outweighed those. That’s just me.

Sickels is just leaving himself some wiggle room in case someone with superhuman abilities was ever born (or arrived from a distant, dying planet in a spaceship that crash lands in small town in Kansas) and took up baseball.

Did Pedro in 1999 have four “plus-plus” pitches?  He didn’t have a cutter at that point, and I don’t think he used a slider more than once or twice a game.  If only he didn’t have this silly rule, I’d be very impressed with his grade variation.  Just give Strasburg the elusive A+, and he’s got a real nice distribution of grades.

BTW… wouldn’t the ideal scouting system be based more on draft rounds (i.e. equally-populated tiers) rather than normally distributed populations?  Given that teams typically get 1 pick per round, at regular intervals, I feel like having that massive number of “C” players doesn’t really help me decide between the one that’s a 9th rounder and the one that’s a 29th rounder.

With this in mind, his system makes a lot of sense if he works for the Yankees or Red Sox, since they’ll presumably never have the draft pick that will allow them to get an A+ player anyways.

Let’s cut to the chase - the real reason he doesn’t give out “A+” is because he doesn’t want to be on the hook for giving out the highest rating, opening the door for a failed prediction, less credibility, hence lessening his product. It’s simply risk aversion. Since there’s always some inherent risk in ANY prospect, he can justify it to himself and everyone else, giving Harper and Strasburg etc A’s (I assume he gave them A’s...)

For his (and his readers’ sake) he should delineate what his grades actually mean. Is an “A” the top 5% when summing all applicable scouting categories? What do the rankings mean?

I agree 100% that practically (or literally) never giving out “A+” is detrimental to his entire mission statement as a “prospect grader”.

If Strasburg is not an A+, no one is. There won’t be a better prospect in my lifetime.

Harper and A-Rod should also get A+. Harper is doing something real special.

15/pm--Harper is a great case for why not to bandy out A+ grades. The power is elite. But the makeup is poor. And with his frame he could look--and run, and field--like Adam Dunn in the near future. Would you make a case for Adam Dunn--at his peak--as an A+? Harper’s an A. Just like Trout is an A. And they’re the two best prospects in baseball.

I don’t think there has to be an A+. If 100 people take the same 100-question test and 5 of them miss 5 questions each, but one person misses only 1 question, 96% overall and a 1% edge over the next-best bunch of scores, to me, is still a good solid A, not an A+.

In MLB, you can make the case for never giving out an A+ to someone out of high school.  That’s because an MLB team doesn’t have a history of allowing this to happen.

Strasburg was out of college, which makes a huge difference.

You can draw a pretty bright line of not giving anyone an A+ unless he’s out of college or hit AA.

You might miss say Doc Gooden’s season.  And you might miss Felix.

But at the least you give yourself some opening for the others.

And yes, I think giving out an A+ will put you in the cross-hairs.

"If Strasburg is not an A+, no one is. There won’t be a better prospect in my lifetime.”

Griffey. A-Rod. Prior. Harper. Three were once in a lifetime prospects before Strasburg, and one… a year after.

I would be comfortable with the following definition of an A+: “An A+ prospect is a high schooler who is ready to start in the major leagues.”

Also, I think there are A+s in hockey and basketball.  LeBron, for example.  Moses Malone (though he was 19.) And tons in hockey - certainly Gretzky, Hawerchuk, Crosby and Yzerman as young 18-year-olds, and Lemieux as a slightly older one.

Cut and pasted from Sickles site;

QUICK PRIMER ON GRADE MEANINGS:

Grade A prospects are the elite. They have a good chance of becoming stars or superstars. Almost all Grade A prospects develop into major league regulars, if injuries or other problems don’t intervene. Note that is a major “if” in some cases.

Grade B prospects have a good chance to enjoy successful careers. Some will develop into stars, some will not. Most end up spending several years in the majors, at the very least in a marginal role.

Grade C prospects are the most common type. These are guys who have something positive going for them, but who may have a question mark or three, or who are just too far away from the majors to get an accurate feel for. A few Grade C guys, especially at the lower levels, do develop into stars. Many end up as role players or bench guys. Some don’t make it at all.

A major point to remember is that grades for pitchers do NOT correspond directly to grades for hitters. Many Grade A pitching prospects fail to develop, often due to injuries. Some Grade C pitching prospects turn out much better than expected.

Also note that there is diversity within each category. I’m a tough grader; Grade C+ is actually good praise coming from me, and some C+ prospects turn out very well indeed.

Finally, keep in mind that all grades are shorthand. You have to read the full comment in the book for my full opinion about a player, the letter grade only tells you so much. A Grade C prospect in rookie ball could end up being very impressive, while a Grade C prospect in Triple-A is likely just a future role player.

Philly/9,

Thanks for that. I’ve been thinking of doing something similar for years, but kept putting it off. Now I don’t have to

Basically I don’t want to give out an A+ to someone because I don’t want to look really freaking stupid if the guy doesn’t pan out. So whoever said I was “risk adverse” about the topic...yeah, that’s a fair assessment.

Yes, this is the real John Sickels.

It might happen someday. I wasn’t writing back when A-Rod or Griffey were prospects, and they might have gotten A+s, but it is hard to say in retrospect.

Someone mentioned the grade clumping in the B-/C+ range. That is often the thing I struggle most with, distinguishing between prospects in that range.  I spend more time agonizing over guys in that grade range than i do about the obvious A/A-/B+ guys.

Tango/10 - What population are you assuming has the normal distribution
here?

If the distribution is of baseball talent of, say, 20-30 year old males,
then simply being in a big-league system should put a player at least 2
SD above the population mean. If the population is of simply those in a
MLB system, it’s less obvious to me that the normal distribution would
be right.

Put another way, Tango’s distribution starting at 0.5 SD implies a total
population of about 1600, and starting at 0 SD he’s implying a
population of 988. Those numbers may be about right for counts of
“prospects” in affiliated baseball, but they’re quite low if you’re
looking at the total population, which seems more likely to follow a
normal distribution.

I’ve tinkered with larger populations, and smaller SD steps further from
the mean, to see whether the dropoff in theory would be different.

First I picked 366000 population, starting at SD 3-3.25 and in 0.25 SD
steps. This gave me this:
283 (240 C+)
126 (92 B-)
53 (85 B)
21 (54 B+)
8 (16 A-)
3 (7 A)
1 (0 A+)

I’ve seen an estimate that there are about 150,000 HS baseball players
in the U.S., so the 366000 number (chosen so that the sum of players >=
3 SD is 494) is in the ballpark of those who actively try to play
baseball in an age range where they might in theory become prospects.

Another population I tried was 15,600,000, which is about the total
number of males in the U.S. near the age of baseball prospects (say,
18-25). This time I start at SD 4-4.25, and again use 0.25 SD steps:
327
114
37
11
3
1
0

The larger the population, the sooner the distribution flattens out, and
the more players receive the lowest grade. In this last case, we’d now
no longer expect any A+ players (I rounded down from an estimate of
0.23).

While I’m sympathetic with Mike/13’s suggestion that he’d rather see
rounds than standard deviation cohorts, it’s harder to make clear
distinctions in talent level between players that are not as good. These
distributions likely more accurately reflect the skill distributions of
prospects. Sickels’s distribution seems like a hybrid: he’s quite
reluctant to give out As, but he gives out more Bs than a normal talent
distribution would justify, assuming his grades should correspond to SD
partitions.

John/23 - Thanks - it’s good to know you agonize most over the C+/B- distinction. That is consistent with the distribution of talent following a right tail of a normal distribution.

Geoff,

The talent distribution I use for MLB talent is here:
http://www.tangotiger.net/talent.html

However, that’s if you know the talent of the player.

In this case, there’s alot of uncertainty, so, that should compress the range.

I just played with numbers to get what I showed.  I can get a better estimate if people cared enough, but I was more interested in just getting something quick out there.

If someone is an A+ prospect by this measure, then that player should be and likely would be already playing in the major leagues.

Tango/26 - Thanks for the link! Amusingly that shows MLB level talent at a bit above 4 SD from the mean, in the ballpark of of the range I used in my largest population estimate (also a quick stab at the problem).

My comment on the last chart is that it’s clearly skewed to the left, and isn’t itself truly normal. But it’s certainly much closer to normal than the distribution would be if you counted each player at 1, rather than weighting by playing time.

And to tie back to this thread, prospects aren’t quite the right tail of the skill distribution - that’s MLB level stars. The prospects are a shade to the left of that. As Geri/27 notes, if the prospect is *that* good, he’s probably just playing in the majors instead of still being just a prospect.

At what point would you have rated A-Rod as an A+ prospect?

Before the 1994 season he was the #1 pick in the country, had signed too late to play pro ball.  While the scouts most certainly loved him, I don’t think there was any certainty as to how great he might be.

Before the 1995 season, he was an 18 year old who moved so quickly that he played 32 games in AAA, hitting 311/359/588.  He was overmatched in the big leagues though, getting 54 AB and hitting .204, all singles.  Is that enough for an A+?  Maybe.  It’s also your last chance to rate him as a prospect, he didn’t play especially well in 1995 but accumulated enough service time that after that season he had a full year in.

To rate Griffey as an A+ you would have had to do it after the 1988 season.  As an 18 year old he hit 338/431/575 in the California League.  On the plus side he’s very young for the league and that is outstanding performance.  On the down side, it’s only 58 games (also played 17 at AA) and coming in a hitter’s league.  Also a superstar by the scouts as the #1 player taken in the draft.

To say Griffey should have been an A+, based only on what was known pre 1989, I’d want to know if his performace was unique, or if other players, now long forgotten, had similar strong starts to their careers.

Evan Longoria was signed to a 9-yr 40MM$ deal days after making his debut.  The Rays thought he was A+.

***

Geoff: if we KNEW the true talent of each prospect, then it WOULD have a distribution in the 3-4 SD range.

The problem is that there is an uncertainty in the estimate.  So, a guy that you think is 3.5 is actually some distribution of 3-4 SD.  A guy you think is 3 is some distribution of 2.5-3.5.

So, you’d have to model that.

And I just threw it out there that it might be proxied by starting at mean=0 or 0.5 and going right.

Longoria:

His 2007 minor league performance was great, but not anything above other great 3B prospects.  Compare his season to what Alex Gordon did in his first year in the minors.

That contract has been often used as the standard for team friendliness around here. You can make a better case that it means Longoria had doubts that he would become a superstar.  For the Rays, once they had an idea that he would be willing to sign that contract, they would have been silly to pass it up as long as they believed he would be even a bit above average.  Remember only 17 million of it is guaranteed.