THE BOOK--Playing The Percentages In Baseball

Cory, a lot of your rebuttal (or however you want to term it) to my comments is fair.

Let me state up front that I did not mean what I said as a general critique of MLBAM or MLBAM stats.  There is a reason that MLBAM is widely recognized as the most successful online presence of the major sports leagues.  There is a reason that MLB.tv and Gameday simply kick butt.  You guys do good work.  And by that I mean MLBAM in general and you and Ross in specific.

Like my criticisms of Baseball Prospectus a week or two ago on this site, I think they were fair, but my original comments were made in a very specific context and did not include the whole of my feelings or experience.  I have a very positive overall view of MLBAM and MLBAM stats.  Within that context I have a very few disagreements or bones to pick or whatever.

I think your general approach to improving the quality of the data is very good, and a lot of it goes unseen.  Particularly your work at improving the input of the stringers and getting well over 95% (and increasing) of the pitch data reported is very important and very good.

I appreciate your openness with the analytical community, and I think that benefits everyone, MLBAM and the clubs included.

I can probably think of more things that I like, but let me address some of my specific disagreements/issues.

The neural net is a poor choice for classification.  It was a good place to start, and good for Ross for coming up with it, implementing it, and debugging a lot of issues with it.  That was not a simple task, and he did an excellent job at it, and I don’t mean to suggest otherwise.

However, it was pointed out at the 2008 Summit that the neural net was going to be fundamentally limited in its accuracy as a classification tool.  All the improvements you mention in #2, 3, 4, are simply bumping up against the ceiling that is imposed by the method.  Marv White did a good job of explaining at that summit why the neural net had fundamental issues.  I can go into more detail if you desire.

My issue/claim is not that you’ve sat on your hands with not improving the neural net, it’s that you’ve chosen to stick with the neural net this long.

We’ve enlisted the help of all 30 clubs, as well as from you and perhaps others on this thread, in collecting classification and training data but we’re limited by the accuracy of the source data. This has been a major effort on our part and continues to this day, and we’d be eager to see the results of any community-based results on this front.

Accurate pitch classification has huge monetary value to the clubs, and there are not very many people who are good at it.  I would guess that the clubs have few people if any who are good at it, and if they have them (Dan Fox?), I doubt they’re giving that data to BAM for free.  I’m not sure why anyone would give away such a valuable IP without a commensurate return.

In addition, our responsibilities to the Pitch-f/x system go far beyond pitch classification. As you can probably imagine this is an expensive and resource-intensive system to operate and maintain in 30 MLB ballparks, so our attention can’t always be focused on pitch classifications or any other specific issue, as much as we might like it to be.

Sure.  I recognize that.  That’s why I said I thought you should hire more people, not that I thought you didn’t have enough work to do and were just sitting around on your hands.  Obviously I have an interest in that, too, as I’d be happy to sell my IP to you guys.  So I don’t claim to be unbiased.  But I do think I’m correct.

hat the research community has been exploiting this data and has generated some amazing research from it is an unexpected benefit, but not one that we can allow to influence our overall objectives or priorities.

I can’t believe that the clubs feel that way.  Well, I can believe it.  I know that they don’t trust the PITCHf/x data enough to use it right now.  But the data could be improved to be trustworthy, and then the value to the clubs would be so high that I would think it would influence your overall objectives and priorities. 

Of course, I don’t know how much of your priorities as an MLBAM organization lie with improving your product as an entertainment source to gain revenues.  It would seem to me as an uninformed but interested observer that that is probably your primary focus.  I don’t know where your priorities lie with regards to data collection for the use of the clubs.  So maybe it’s not really your bailiwick that I’m criticizing here, but I think someone associated with MLB should do what it takes to make the PITCHf/x data useful to the clubs.

As for Bloomberg Sports, they can defend their own products, but the suggestion that we’re not trying to improve our data because they are licensing from us is not only incorrect but completely counterintuitive. On the contrary, we have every incentive to improve our classifications—and ALL of our data—to make sure we are provide the best possible product to a major partner, which will in turn enable them to find better acceptance in the marketplace for their products.

I’ll admit that my dig at Bloomberg was out of frustration at their ability to get a wow factor with nice charts while I question the ability to do meaningful analysis without much more accurate pitch classifications.  There is and was no reason for me to think that this would produce an incentive for you to stagnate, and I should not have suggested such.

Btw, it’s nice to see that you still read things here.

Ugh, let me clarify one comment I made before it turns ugly, too.

I know that they don’t trust the PITCHf/x data enough to use it right now.

This comes across as a blanket statement about all clubs, but that’s not how I meant it.  A few clubs have hired PITCHf/x experts and are making good use of the data.  A number of other clubs have looked at the data and identified issues that they felt prevented them from making use of it.  Those are the facts.

I also have a feeling that there is a general climate among the other non-using clubs that the data is hard to use and/or has issues, and there are things that could be done to make the data more usable for them.

Again, as I stated (or tried to) in my first comment, I don’t know whether the responsibility for this falls in your lap, Cory.  It’s a criticism of and simulataneously a hope for the game of baseball.  There may not be anyone’s lap in which the responsibility falls, and that’s okay.  I’d like to see that change, but let me be very clear that I’m not criticizing you or Ross or MLBAM for not making that happen.  You have, IMO, been making very good things happen with your work, and I’m not making a call to derail that or point you in another direction.  I’m making a general call for where I’d like to see the game of baseball go and what I think needs to be done to make that happen. I’m trying to do my part to bring that about. 

Sometimes I get frustrated when it doesn’t happen as quickly as I want.  I am a very up front person about how I am feeling.  I hope that does not come across as a personal criticism of you or your work.  I do not intend it that way, and if I have offended, I apologize.  I recognize that I could be more careful in how I word things.  Sometimes I am surprised that people take the time to read all the way through my extensive ramblings, diatribes, etc.  I hope they are valuable and don’t come across primarily as complaints. 

I’m not perfect either, and my work isn’t perfect.  I don’t expect you to align with my every desire for MLBAM.  I’m sure you have insights into your market and your goals that help you see things that I don’t see.

I also hope that it can be said of me that I am as free and generous with my compliments, or moreso, than I am frank and honest with my criticisms, not just on this topic, but on all topics. 

If that is not the case, Tango, Cory, and the regular contributors here are welcome to call me on that and tell me to be more careful about how I come across.

Hopefully my criticisms are also respectful even if they are direct.  “Crappy” is maybe not very respectful, but it is easier to throw that out than to say 76% accurate, because then people are going to want to know where 76% came from and how it can be justified.  Maybe I owe that level of detail, but on the other hand, damn, this is not my day job yet.  It’s only rarely that I get paid for my opinions about baseball (and I have plenty of them).

Sigh. Who let Mike out of his cubicle this morning? Someone please find him and tell him to get back to work.

Signed,
The Management

I don’t think I’ve ever conceded that neural nets are a bad approach or fundamentally limited.  I did, however, concede that the generic neural net with biasing approach we leveraged had limitations due to the sometimes radical differences between pitchers (submariners give me cold sweats).  When I use data for a single pitcher as training data, the nets typically produce results that are over 99% accurate (testing note: If I use 1000 pitches for training, I’ll break the pitches up into a training set of 750 pitches and an evaluation set of 250.  By “accuracy” I’m referring to the correlation between what the network spits out and what an expert (the source of the training data) claimed).  For instance:

results for 150274: Nathan, Joe
242 out of 244 classified similarly - 99.18033%
CU - Same: 20, Different: 0
SI - Same: 30, Different: 1
SL - Same: 77, Different: 1
FF - Same: 115, Different: 0

Or for a really easy one:
results for 282332: Sabathia, CC
826 out of 826 classified similarly - 100.0%
CH - Same: 153, Different: 0
SL - Same: 194, Different: 0
FF - Same: 479, Different: 0

This season we are introducing a mechanism that will allow us to generate and modify custom neural nets for every mlb pitcher.  There might be some pitchers who cause problems for these new nets, I haven’t made a Washburn net yet, but results are extremely promising so far.  I’ll try to let the community know when we roll some of them out so you can let us know how we are doing. 

I still think neural nets are an ideal choice for classification.  They give you blazingly fast execution speed, are robust to errors in the training data, and relatively easy to manipulate.  Decision trees would give us more visibility into how the classification was made, but they are fragile to shoddy training data.  Other approaches we’ve considered all have problems that fail to address our requirement of classifying every pitch, from every pitcher, in real time. K-means is an excellent approach for offline analysis, but it requires you to have data to cluster against, and know the number of clusters beforehand.... As a post game analysis that can produce training data for new neural nets however...... 

Anyway, I’m really excited to hear your take on our new results.  And I need some serious analysis done to tell me if my Phillies giving up Cliff Lee for Haladay shouldn’t be breaking my heart.

Hi Ross-

Could you provide us data for a pitcher like Bronson Arroyo?

Here is what I get when I plot all the MLBAM pitch IDs for him from last year’s data:

-Dan

Dan,
Will do as soon as i generate a net for him.  I don’t have any training data for him so we need to create some.  And, as we still have a solid few weeks till opening day, our new mechanism for generating training data isn’t quite ready yet .  Anyway, if you have some you want me to test with beforehand, shoot it on over.  I’d just need sv_id, game_pk and your classification type.

Cory - At the 2009 Pitch f/x Summit MLBAM made it possible for me to observe the MLBAM stringer recording the Gameday information.  I was accompanied by Ross Paul.  Ross told me that although he had designed much of the input screen procedures he had never actually had the opportunity to watch a stringer work under game conditions.  We were both impressed at how hard she had to work to record all the information during the limited time available to her between pitches.  As we were leaving, Ross commented that he would have to examine the input process and see if he could make the job easier for the stringers and yet preserve the integrity of the data. 

As I was flying back home from the Summit it occurred to me that if Ross was going to redesign the input process for efficiency, that it might also be possible to include changes that would make the data more useful for analysis.  I contacted Ross and offered my assistance in trying to redesign with both goals in mind, efficiency and inclusiveness, keeping the limitation of having to be in real time and having to be vitually errorless and conform to the arcane scoring rules that have developed in MLB over 125+ years.

Although Ross was enthusiastic over some of my ideas, my offer of direct assistance was turned down.  I subsequently documented all the changes I would have tried to incorporate, and Ross said that they would receive serious consideration when the redesign took place in the off season.

Several weeks ago when I contacted Ross to see what progress had been made on the redesign, he told me that only minor changes had been made because of a lack of manpower to work on the project.  Needless to say I was disappointed.

Everybody at this site and everybody that I have met at MLBAM want exactly the same thing, for Gameday to be the very best that it can be.  And most of us here are fully cognizant that your first priority is your real time reporting of game information to the users of Gameday.  But at the same time Gameday could be every bit as good and probably better than BIS or STATS or any other for pay data provider.  The STATS stringer and the Gameday stringer sat side by side at the game that I watched in SF.  They both had the same view of the on field action and the same time time restraints of recording the information.  And both of their employers depending on getting absolutely accurate data.  So why are the teams left needing to purchase the same information from STATS that the Gameday stringer could provide just as easily?

Neither Mike or I are trying to be critical of Gameday or MLBAM.  I think that Mike would agree with me that Gameday is a very, very good product and that we are lucky to have access to it.  And both of us applaud your participation and the openness and cooperation that you have shown here and in the personal contacts that we have had with you.  We both just want Gameday to be the very best it can be, and if you can find a way to incorporate our assistance I think we can help accomplish that.

Peter, your offer of assistance was and is still is appreciated. The simple reason we opted to not implement any such changes as you suggested for this season is that, rather than continuing with the iterative approach we’ve taken towards improving our stringer app, we are scoping out a complete and total re-write of the app for 2011. We haven’t made any decisions about what the UI will look like at that point but the entire process is under consideration.

Thanks,
Cory

Neither Mike or I are trying to be critical of Gameday or MLBAM.  I think that Mike would agree with me that Gameday is a very, very good product and that we are lucky to have access to it.

Yes.  When I consider what fans of other sports* have to deal with, I pity the fools. 

* With a mulligan for hockey on behalf our of host and a fond memory of that Brandon Wheat Kings game that I attended with my cousin way back when.

I wonder if most GM’s want the publicly available pitch classification improved, especially those who spend a lot of time scouting opponents relative to those teams who prefer to scout via gameday.

“I know that they don’t trust the PITCHf/x data enough to use it right now.”

Each team knows what their own pitchers are throwing.  With the data on PITCH f/x and their own knowledge of what is thrown by their pitchers, they can develop their own classification, and extrapolate it to opponents pitchers.  Those who do the best job gain a competitive advantage.

Pft/11, teams buy their pitch classification info from BIS. It is more accurate than MLBAM’s classification. 

It’s not much of a stretch to assume that Josh Kalk is doing his pitch classifications from corrected PITCHf/x data for the Rays since he was doing that publicly before they hired him.  It wouldn’t be a leap to think that Sheehan and Fox are doing PITCHf/x-based pitch classifications for the Pirates since they were the first guys to do it publicly.  Some other teams or players may make use of the data in isolated cases, but I’m wracking my brain for another team that I think is using PITCHf/x data for pitch classification on a regular basis, and I’m not coming up with one.

Your point is well made, though, pft.  I don’t disagree.

I could be wrong but don’t the pirates use their own form of pitch f/x?

I remember going to PNC and they actually have on one of the scoreboards the type of pitch, velocity of pitch and break.  I don’t know if they just post what the pitch f/x says or they have their own cameras.  I thought I remembered reading an article tho that they had their own cameras.

The Pirates have the same PITCHf/x camera systems installed as in all the other 29 MLB parks.  Some stadiums display the pitch velocity from PITCHf/x and Ross’s real-time pitch classification on the scoreboard.  I know they do that at Minute Maid in Houston.

If they are displaying PFX or BRK from Gameday, I will have to sigh.  The PFX quantity is horribly misleading, and I can’t believe that MLBAM has kept that in Gameday all these years with so many of us telling them that it’s a bad idea.  BRK at least isn’t that misleading, but if you are already telling someone whether the pitch was a fastball or a curveball, then BRK is not really helpful.

They definitely display some kind of break data at PNC, but I think it might be horizontal and vertical.  I’ve only been there once, last summer, but I distinctly remember thinking how cool it was that they displayed the pitchf/x numbers live.  I wish the Nats did that.

If they are displaying pfx_x and pfx_z, whatever they happen to be calling them, that is way cool.  I would love that if I were at a game there.  However, I wonder how many people in the park have the faintest idea what those quantities mean.

Yeah I remember seeing it and thinking wow that is way cool.  I’m sure that 98% of the fans in there had no idea what those numbers meant tho other than pitch type lol

now that I think about it more it was probably pfx and break which isn’t great but still interesting they would put that up anyways.

I think that if it was pfx_x and pfx_z I would have definitely remembered.

I’ve compiled a pitcher scouting database based on the excellent work of Sven at 60ft6in.com. It is referenced by mlbid.

Notably, for K-means pitch classification analysts, it has a field with number of pitches in the pitcher’s repertoire.

If Ross is doing the hard work of getting NN’s set up, and if they are working as well as he says, I think the least we can do as a community is assist him in classifying the tough pitchers.

This database will help.

http://blog.rotobase.com/2010/03/pitcher-scouting-database/

The scouting reports aren’t perfect, but I think this is a very good start. Any community additions to this database would be super.

Now I’m going to go lose myself in rapache for a while.

Josh,
That’s awesome and amazingly helpful.  Thanks!

re: #17

Yes, the boards down the first and third base lines last year displayed pfx_x and pfx_z in addition to initial velocity and velocity at the plate. This was implemented by the scoreboard vendor.

And yes, there were many many questions from fans, broadcasters, front office folks and the like. Shortly after the season started when those questions rolled in we worked with the vendor to adjust the pfx_z value to make it relative to a fastball thrown at the same velocity with backspin. That helped quite a bit but of course there were still questions. There might be some changes to it this year although I don’t know for sure.

Hi, Dan!  It’s good to see you around.  Thanks for the clarification.  That is very cool that the scoreboards are showing spin deflection.

Mike,
Most teams don’t care or do not want a better mlbam classification algorithm.  8 or 9 teams couldn’t care less about this data, and most of the others probably want to create some kind of competitive advantage with their own breed of pitchfx and hitfx data.
I know of at least two other MLB teams (other than the ones you mentioned) with independent pitch classification schemes and sophisticated pitch and batted ball analysis. There are more teams I don’t know about I’m sure! They have no incentive to advertise their work, so I think we tend to underestimate front offices’ statistical capabilities. But most teams have at least one intelligent person with technical skills working for them. And these people have access to BIS data, coaches’ reports, and scouting reports to help with the task!

Re Dan#22 (and I agree with Mike Fast, good to see you back!):  Back in the early days of pitchf/x, there was lots of discussion on the blogs about “movement” (pfx values) vs. “break”.  I won’t try to repeat any of that here.  However, I think the general consensus was that pfx values are what baseball analysts want to see but the break values are more intuitive to players, coaches, fans, and broadcasters.  Taking the pfx values and renormalizing so that they are relative to a fastball (as Dan reports in Pitt.) is a step towards bridging the gap between pfx and break.

Now we all need to figure out who “Anonymous” is (#24)!

I’m pretty sure I know who one of the other teams is that Anonymous refers to in #24.  I should have thought of them at first.  However, since I don’t know that they’ve made any public moves that would indicate it, I won’t mention their identity here.

There’s another team which I know has very sophisticated analysis capabilities but I also know is not using PITCHf/x data in any extensive fashion.  I’m struggling to think of who else might fit the description:

I know of at least two other MLB teams (other than the ones you mentioned) with independent pitch classification schemes and sophisticated pitch and batted ball analysis.

We are starting to create the custom neural nets for each pitcher and I have some fun results for you.  Last season, we weren’t the best with Santana.  We’d mix up some SLs and CHs, some FFs and FTs, and worst of all, call some CHs and FTs.  For instance for the August 9th game we classified his pitches like so (FF=orange, FT=blue-gray CH=green SL=blue):

Using data from alternate games we trained a custom net for him and then regenerated our outputs resulting in:

pretty cool, right?  FYI, the other pitchers in that game do not have custom nets yet and we haven’t regenerated other games for Johan yet either.

Awesome!

Dan,
As requested here’s a before and after for Arroyo.  Guy throws everything!  Our custom net has him throwing 7 pitches!!
Before (SL=dark blue, CU=gray, CH=dark green, FA=light blue, FF=orange, FC=light green)

And the after (CU=Gray, SL=bright blue, FS=yellow, CH=dark green, SI=dark blue, FF=orange, FC=light green)

Just updated my database today.  MLBAM shows Wakefield with 11% fastballs, mostly knuckleballs, and a few curves too.  Big improvement over last year showing all knuckleballs.