THE BOOK--Playing The Percentages In Baseball

Yes, that’s exactly what I was getting at. Well said.

Yea, this is the best summation of the entire SIERA inkstorm that I’ve read so far.

Agreed, you and Patriot put into words what was lurking in the back of my mind. The “throw everything (every term, squared term, interaction) at the wall and see what sticks” method of regression analysis always seems not very well-grounded, and you’ve explained why: it’s not based on anything in reality.

Still I know that it’s tempting, because I do it myself when I’m fooling around with data!

The thing with SIERA is that when you read Matt’s explanation, he DOES breaks things down, and tries to tell you why he’s using each component.  In a sense, while he tested the “let’s throw everything against the wall”, he then tries to figure out the reason why we might be seeing certain things.

It’s just the next step that’s the issue.  Should he simply merge it all into one formula, or break it down like Patriot is suggesting?

Take for example the regression of events to team runs scored.  The run value of the double is something like +.15 above the single, which is not only ridiculously low, but completely wrong.  But, what if the double is capturing some hidden relationship (i.e., teams that hit alot of doubles are slow runners).  Then what?  Well, either you do something like:

hitting = .50*singles + .80*doubles +…
baserunning = -.15 * doubles

OR, you do:
offense = .50*singles + .65*doubles +…

Now, which one is clearer?  And, what if you finally improve the baserunning equation to the point that you don’t need doubles?  Do we want to see the second equation change from .65 to .80 because some OTHER variable was introduced?

Or, do we prefer like in the first sets of equation?

This really comes down to mostly a question of presentation.

***

We can be just as harsh on Baseball Prospectus showing it’s percentile forecasts.  I would guess that if I were to quiz the BPro readers, 90% of them would fail my test.

Take for example Felix’s forecast:
http://www.baseballprospectus.com/card/card.php?id=HERNANDEZ19860408A

The data under the W/L columns go from at best 16-6 to at worst 15-7.

And if I were to ask: what are the chances that Felix will be .500 or worse, given what you see anywhere in that “2011 Forecast” section.  What do you think they will answer?

Or look at the data under HR, where it is at best 15 and at worst 15.  And I were to say: “what are the chances he will give up at least 16 HR”.  What do you think they will answer?

I know EXACTLY why the data shown there is shown in that manner.  I don’t need the explanation.  But at least 90% of BPro readers would, and 99% of mainstream readers who stumble onto BPro would need the explanation and would fail my test.

Sometimes, the presenter has to make a choice that may not be the most appealing, but it is the easiest.  SIERA is like that, and the percentile forecasts are like that too.

Yea, you don’t need a PhD in Math to know those BP percentiles are completely broken - or at least the ones on Felix’s page. Who could publish those and think “yep, 15 HR… looks good!” Pretty funny actually.

Lee: it’s not necessarily that they are “completely broken”, but that the presentation of it is indecipherable for the majority of readers.

You come in cold, to that page, and you have no choice but to laugh at what you see.  As Danny Vuko would say: “Ha.  Ha.  Ha.”

But, what if instead what you see is an implied “average” for the ERA forecast at that line?  So, what it’s really saying is this:

“The 90th percentile forecast applies ONLY to ERA.  All the other columns are the AVERAGE you would get, that would allow you to produce that ERA.”

Basically, except for ERA, everything in that chart is “reverse-engineered”.

So, you go from a chart that is completely ridiculous to a chart that is completely rational… and all you did was change your glasses.

***

If for example you wanted to know the 90th percentile forecast for Felix’s HR, you will NOT find it anywhere in that section.

Re pitcher metrics- it seems like if you’re going to address the interdependence and non-linearity of the inputs and how they differ by pitcher type, you’re going to end up with an equation that’s pretty complicated regardless of whether it’s derived via regression or by using a run creation model.

Personally, I like FIP for the reasons mentioned- it’s simple and its limitations are pretty clear.  But there is an argument to be made for something like SIERA.  Namely, folks are inevitably going to over-interpret and over-apply the metric, so you may as well knock yourself out trying to make it absolutely as accurate as possible.

I agree that SIERA has a place at the buffet table.  Everybody loves the FIP chocolate, but some also like the SIERA escargots.

Can someone point me in the best place to read about the Siera imbroglio and criticism that the metric is intelligible?

John/9,

Matt’s five-part series launching his modified version of SIERA at FanGraphs is here:
http://www.fangraphs.com/blogs/index.php/new-siera-part-one-of-five-pitchers-with-high-strikeouts-have-low-babips/
http://www.fangraphs.com/blogs/index.php/new-siera-part-two-of-five-unlocking-underrated-pitching-skills/
http://www.fangraphs.com/blogs/index.php/new-siera-part-three-of-five-differences-between-xfips-and-sieras/
http://www.fangraphs.com/blogs/index.php/new-siera-part-four-of-five-testing/
http://www.fangraphs.com/blogs/index.php/new-siera-part-five-of-five-what-didnt-work/

Colin’s article announcing that the original version of SIERA would no longer be carried at BPro is here:
http://www.baseballprospectus.com/article.php?articleid=14603

Patriot’s post on his blog on run estimators is linked above.

Graham MacAree posted an article at Lookout Landing about his thoughts about sabermetrics.  It has a broader aim than just addressing SIERA, but it’s part of the discusion:
http://www.lookoutlanding.com/2011/7/20/2285655/the-problem-with-sabermetrics

Tango’s thread on new SIERA is here:
http://www.insidethebook.com/ee/index.php/site/comments/siera_updated/

David Appelman’s defense of publishing SIERA is here:
http://www.fangraphs.com/blogs/index.php/your-sabermetric-choices/

Matt has a mailbag thread for SIERA questions here:
http://www.fangraphs.com/blogs/index.php/siera-mailbag/

Mike/10 was automatically blocked for moderation and is now open.

Graham’s article was devoid of any real examples.  His overall summary opinion, which seemed far-reaching, lacked evidence to support that. 

Normally, I’d call it bullsh!t, but the article itself didn’t read as bullsh!t.  A weird article to read basically.

If you haven’t followed Graham on Twitter or read his previous work on Lookout Landing, it would probably be a hard article to interpret.  As evidenced by how people on BBTF and B-Ref totally missed what he was saying.

He does explain a few more things in the comments.

I debated whether I should even list Graham’s article as part of the broader discussion, as I don’t think it was necessarily intended that way.  But it seems to have become that, so I listed it.  Plus, I agreed with a lot of what he had to say.

Someone else had written an article a week or three ago about how sabermetrics has never been in better shape.

The level of sabermetric signal is extremely high.  If there is a problem, it’s the lazy reader who doesn’t know exactly where to go and what to listen to, or even how to listen.

Graham’s article therefore was one-sided.  He may have made some fine general points, but his overall conclusion had no evidence to back him up.  (It reminded me alot of Gary’s “Baseball Analysis is dead” piece.  That one was myopic.)

Like I said, it was a non-bullsh!t bullsh!t article.  The equivalent of the non-intentional intentional walk!

I shouldn’t have said “lazy”.  The correct term is “unmotivated”.

Tango, did you ever derive FIP co-efficients using regression?  If so, was the equation very different, or basically the same as the one you came up with?  Thanks.  I ask because I’m curious whether regression is really an inferior method, or just different and perhaps less intuitive.

I originally used regression.  But I derived it later on as well. 

Regression is not necessarily inferior.  It’s a first-step, if you don’t know how things work.

Basically, there’s a time and a place for regression.

So I look at what Matt did, and I can see exactly the thought process.  It’s “you know, there’s no reason to think that the impact of walks (or whatever) on RA is linear or independent of the other variables, and every reason to think that it isn’t.  Let’s regress the bejeezus out of all this data and figure out which terms are significant and why”. 

Now, once you do all this, you may find that the improvement isn’t worth all the added complexity.  Or, you may think that any improvement justifies the effort.  Whatever your preference.  But it doesn’t strike me as a methodological issue.  The methodology seems just fine.

Pierre: I agree with your first paragraph.

The next step would have been to then build the findings into a run modeler.  If you insist on complexity, at least put it into a working complex system. 

This is the point I am trying to make.

We could make Runs Created a regression equation similar to what Matt is doing, with quadratic terms, multiplying some of the components (HR x BB), etc.  You’ll have a very long and complex equation.  And, maybe you can even justify each and every component.

But, it doesn’t fit within something that we know: how runs are scored in baseball.  We know exactly how that works.  You get runners on, and they move over, until the inning is over. 

If you insist on complexity, at least work within whatever thing you are actually doing.

Hence, the reason that BaseRuns is far far far better than any regression-based runs equation you can come up with.

Could you get at some of these issues using the model that underpins Fip or wOBA?  Could you test the hypothsis “walks are less bad for ptichers with good control becuase they are issued more strategically” or “HRs are a bigger issue for pitchers with really good k/bb ratios because there aren’t going to be many long rallies”?  Off the top of my head, it seems like it would be really tricky.

Right, I wouldn’t use FIP or wOBA for that.  Let’s say that this is a true statement:

walks are less bad for ptichers with good control becuase they are issued more strategically

The next step is: how to model that.  Well, why not model it within a model that approximates run scoring?  Now, you are correct that we just couldn’t use BaseRuns, because BaseRuns presumes random distribution of events across base/out states.

Therefore, we need a BaseRuns-type model that includes sequencing.  Colin’s FRA purports itself to do that.  I’m not sure that it actually does do that.  But in any case, that would be a good next step: to have a BaseRuns model that includes the base/out state.

Perhaps I am being too harsh on Matt’s model, now that I think more about it.  After all, having a base-out version of BaseRuns would be fantastically complex.  So, Matt’s equation, while appearing fairly complex already, may in fact be “simple” compared to the complex system that I am envisioning.

So, that is a very good example you chose, because it needs to be addressed within a model that does not keep the frequency of events static across base/out states.

Good job!

Thanks.  I really like your work and appreciate your willingness to answer questions about it.

Mike/10—perfect and thank you