THE BOOK--Playing The Percentages In Baseball

And Dan Fox also shows his comp to King Felix:

http://www.baseballprospectus.com/article.php?articleid=7346

Mike - Your charts seem to have the 4 seamers and 2 seamers reversed.  Is the dominate fastball pitch in this data set the 2 seam (sinking) fastball or the 4 seam (rising) fastball?  Much of MLB’s (Sportvision’s) fastball problems seem to be the result of having one too many categories.  I couldn’t find a translation between your number codes for your PitchID’s in your Excel files to your Pitch descriptions.  Also, they didn’t show an algorithm that created your PitchID’s.  Are you doing it by hand or do you have an algorithm that you just didn’t want to reveal?

Peter, my pitch classification was done by hand.  I don’t have an algorithm.  I’d love to have one; I just haven’t found one that I’m satisfied with yet.  Either that or I’m not smart enough with statistical clustering tools to make one.  It definitely could be that.

I think MLBAM’s categories are okay.  They have FF and SI, presumably for four-seamer and two-seamer, respectively, and then FA I assume is the catch-all, we-couldn’t-tell between four-seam and two-seam category, which is fine by me.  I tend to assume a fastball is a four-seamer unless I have evidence to the contrary.  MLB is making a similar, but more agnostic, assumption.  They don’t seem to put things in the FF bucket very often.

The dominant pitch in this data set is definitely the four-seam fastball.  Lester is the only pitcher of the group that I know definitely throws a two-seamer.  I don’t see that I have anything reversed.  Maybe I’m missing something.

Mike - It was my misunderstanding of the FA as the catchall category of fastball.  Since you had no catchall you had a large number under your four-seamer category where MLB had a small number under FF.  If MLB put their SI in your two seamer and their FA and the one FF in your 4 seamer, then it looks like agreement on the fastball would improve.  Do you have characteristics that would distinguish a two seamer from a sinking fastball or does it make sense to combine those two categories?  It would seem to make more sense to ask the batter what pitches he is seeing than to ask the pitcher what he is throwing since the batter is the one who has to be able to see a functional difference between the pitches.

Perhaps we should consider the current state of Sportvision’s pitch classification as Beta testing.  After all, if they refine their algorithm or you or Josh or anyone else comes up with a better one they can always go back and apply it to the past data; both this years and last years.

Its also helpful to remember that the primary client for this data is MLBAM’s Gameday and the use is to provide information on the screen for fan consumption.  If we want analysis grade information we may have to provide our own algorithms for it.

I’d love to have the access to be able to ask either one, or both!  Very rarely do I run across comments by batters about what pitch type they saw.  Such comments by pitchers about what pitch type they threw in a given situation or about their general repertoire are fairly frequent.

In grading MLB’s fastball accuracy, I was generous to them regarding their FA category, counting it as correct for both two-seamers and four-seamers.  Only when they specifically tried to differentiate for SI and FF did I grade them on how well they differentiated.

In my opinion, for most purposes, it makes sense to combine two-seamers and sinkers in one category.  It’s hard enough to tell two-seamers from four-seamers.  My understanding from conversations with SirKodiak about grip is that a sinker is a special subset of two-seamers where the thumb is tucked farther down under when gripping the ball, giving it more sidespin.

I have not rigorously studied the issue since currently my catalog of images of pitcher grips is very small, but my wild guess is that there is a continuum of spin axes that different pitchers put on their two-seamers, and there is not a sharp line between two-seamer with the thumb out and sinker with the thumb under.

One rule of thumb that I use for whether to classify two pitches with similar characteristics but slightly different movement as different pitches is whether any given pitcher throws both variations and views them as different pitches.  Many pitchers throw both a four-seamer and two-seamer as distinct pitches.  Some throw both a splitter and a regular change-up, and some throw both a cutter and slider.  So those are all distinct pitches, in my opinion. 

I’m not aware of anyone who throws both a two-seamer and a sinker and calls them separate pitches.  Greg Maddux intentionally varies the spin on his two-seamer, but I’m not aware of him saying that he throws two different kinds of two-seamer, just one pitch with many slight variations.

My second, more practical rule of thumb is that if I cannot find any way to reliably tell the pitches apart by speed, spin, or movement, I’ll consider them one pitch for my purposes.

When you get into circle change versus straight chagne, or two-seamer versus sinker, I have no doubt that these are different pitches thrown with different grips, but I can’t practically tell them apart in the PITCHf/x data.

Peter/#5, I had privately heard some comments about the MLB pitch classification algorithm prior to its release which I allowed to raise my expectations for it too high.  I was hoping it would perform better than what was already available on the Internet for free.

Peter/#6, you make a good point, but do you think the average Gameday-using fan is happy with 80% accuracy?  Which really means that some days with some pitchers they will 95-99% accurate and other days with other pitchers they will be <50% accurate.  I think you could come pretty close to 70% accuracy just by watching the stadium radar gun alone.

MLB also has some difficulty with your underhand-throwers and side-armers.  I noticed that they classified Brian Shouse’s “fastball” as a curveball because of its downward movement--similar to a lefty’s curve.  Note that this is exactly how Walsh described it in his THT Pitch Identification Tutorial.

They probably need to think about using the release point data to help this kind of situation.  I don’t know if this is how Josh Kalk’s algorithm handles this type of situation or if there’s some kind of hard-coding that goes into his algorithm for certain pitchers.

This is why it matters to the average Gameday user that the pitch classification is accurate (game commentary pulled from a Yankees blog):

I’ve got to follow the game on MLB Gameday. They indicate Wang is throwing cutters.
Is he really throwing a cutter?

Labeling a slider as a cutter is not just a problem for analysis grade data recording.  It’s a problem for fans who want to understand their pitchers.

You’ve convinced me!  Now all you have to do is get to work on your algorithm to solve the problem.  That would make a nice presentation at the seminar.

The pitches don’t really NEED names, they seem to have them just to better describe the action to the fans (we know what a curveball is, but we generally don’t know what a 76 mph pitch with 3.45in of movement in x and -3.73in of movement in z means). So instead of creating this algorithm to guess what the pitches are, why don’t they have someone label them by hand? Most fans know the pitches that pitchers on their teams throw, so why not have someone (or a few people) type in the type of pitch thrown?

To use Mike’s example in #10, every yankee fan can tell you that Wang throws a whole lot of sinkers, and a few sliders and a few change-ups. If we had a few people inputting the names of pitches into the system by hand then we wouldn’t see cutters coming up on the screen.

This way, when you see a curveball on gameday, you would know it’s not a mis-labeled slider. I think this would be MUCH more accurate than any algorithm that anyone could create.

Last year, someone had suggested to me that I do a Fan survey similar to the others I do, where I ask the fans to tell me what pitches each pitcher throws.  I was skeptical of the idea, since I figured we’d be able to come up with a pitcher’s repertoire based on the speed thrown and (implied) spin imparted (direction, rpm).

Now that we are where we are when we are: Is there any reason to run a Pitcher Repertoire survey?

No.  Mike Fast’s work shows that the difference between some of the pitch types is subtle, but that it is possible to distinguish between pitches by speed and spin type.  It just isn’t possible yet to do it automatically.  If the differences are too subtle to design a metric to distinguish them from the data provided by the Sportvision cameras, then the differences would also be too subtle to pick up by the human observation of the average fan.

I would argue that most fans do not know what most of the pitchers on their teams throw.  Take Aaron Gleeman as an example.  He’s a smart guy, and I don’t think many would argue against him as an expert on the Twins.  However, he can’t figure out what Francisco Liriano was throwing yesterday:
http://www.aarongleeman.com/2008_04_13_baseballblog_archive.html#9035114725817217916

Using a variety of sources, he still got it wrong.  With the PITCHf/x data, I can clearly see the difference between Liriano’s slider and change-up, no question, and even begin to tease out the difference between his four-seamer and two-seamer.

The pitches probably don’t need names for some applications, but for other applications knowing the names makes all the difference.  For those applications what we’d really like to know is how the pitcher gripped the pitched and the biomechanics of what his hand did on release, but the pitch name combined with the speed and movement data can serve as a pretty good proxy for that.

Just to be clear what I am suggesting and not:
1. I am NOT suggesting that we have fans introduce what they think was thrown, pitch by pitch

2. Someone else WAS suggesting if we can come up with an inventory of what a pitcher throws

The purpose here is to have a pitcher linked to 3 or 4 or 5 pitch types, so that when a researcher like Josh Kalk and Mike Fast and BIS and MLB.com runs their algorithm, that the Inventory would act as a initial sanity check to make sure that their results correspond to expectations.

So, the question is really beneficial only to the handful of people who look at the PITCHf/x data to gives pitches names.  If BIS is saying Brian Bannister throws some pitch 10% of the time, and Bannister himself says he’s never thrown the pitch, then the algorithm needs tweaking (or perhaps someone needs to tell Bannister that the pitch he’s trying to throw is not at all the pitch that he is throwing).

Does it have value, therefore, to have an inventory of every pitcher’s pitch?

While it’s true that I struggled to identify Liriano’s pitches at times yesterday, my default/fallback for each pitch was MLB.com’s labeling.

You’re saying that it’s easy to distinguish changeup from slider using PITCHf/x and I’m sure that’s the case given how much work you’ve done using it, but does that mean MLB.com’s system isn’t able to do that distinguishing during games?

Since 95 percent of “my” identification of Liriano’s pitches yesterday simply came from MLB.com, then I’m guessing that means what PITCHf/x says to someone who knows what to look for isn’t the same as what PITCHf/x says on the Gameday screen?

OK, one other thing.  In the few instances where my labeling differed from what MLB.com Gameday classified a pitch as, it was because Liriano himself or Ron Gardenhire identified the pitch differently.

In other words, 95 percent of the time I just went with whatever MLB.com labeled a pitch, but on a few occasions MLB.com called something a changeup and after the game Liriano/Gardenhire specifically said that the pitch was a slider.

Does it have value, therefore, to have an inventory of every pitcher’s pitch?

I think so.  For more reasons than just being able to correctly identify them at game time.

Aaron, in Liriano’s case, the component of the spin rate in the x-z plane is a clear indicator of slider or change-up.  A slider like Liriano’s has most of its spin around the direction of travel, like a spiraling football.  This makes the spin in the x-z plane very low.

In Liriano’s case, the spin deflection (pfx_x and pfx_z) is also quite different between the slider and the change-up.

I don’t know the details of MLBAM’s pitch algorithm, but it appears they don’t calculate the spin axis or spin rate.  They mucked up the classification of Liriano’s slider.  Liriano actually threw 23 sliders, and their algorithm labeled those pitches 5 sliders, 13 change-ups, 3 curveballs, and 2 cutters.  I have no idea how an algorithm could realistically call them anything but a slider unless it was primarily classifying based on speed or had something else set wrong.

Using Pitch F/X isnt it rather easy to distinguish between changeup and slider.  The changeup moves in the same direction as the fastball, whereas the slider moves in the opposite.

I can see not being able to tell a slider from a cut fastball since they both move in the same direction (I think), but I dont get the changeup and slider not being able to differeniate

Mike, do they still report a confidence level on the labeling of each pitch within the data?  If so, it would be interesting to see what confidence level it gave for each Lirianos sliders.

I had heard somewhere that MLBAM was going to use some sort of neural net algorithm for pitch ID.  I only know a little about such algorithms, but it’s my understanding that they can be very sensitive to the training they are given, as well as the number of variables that are given as inputs.

Ike, they do report a confidence level.

For Liriano, the confidence level for fastballs was 0.83 to 0.96, and all were identified correctly.  For change-ups, the confidence level was 0.70 to 0.86, and all were identified correctly. 

For sliders, the confidence level was 0.37 to 0.48, and it wasn’t really any higher for the sliders they labeled correctly than for the ones they didn’t.

Heck, based on that, you could almost ID the pitches based solely on the confidence level....

Does anybody know anything about BIS’s pitch classification? As far as I know, BIS has folks watching on TV and classifying pitches visually. Is that correct, does anybody know? 

Mike, have you (or anybody) done a comparison of pitch-f/x vs. BIS?

John, a BIS person said as much (video only).

John,
There’s this from David Appelman, with the response by ultxmxpx and me:
http://www.fangraphs.com/blogs/index.php/hudson-perez-bis-pitchfx/

My general opinion is that BIS does a pretty good job.  They have quite a bit of trouble with the slider/cutter distinction.  They have occasional trouble with the splitter/changeup distinction (I do too, even from PITCHf/x), and they have rare trouble with the slider/curve distinction.  But overall, they do well.