THE BOOK--Playing The Percentages In Baseball

The test was against regular ERA-- NOT park-adjusted ERA. So SIERA and xFIP should have been punished. I could have multiplied them by their park factors, and presumably, they would have done better.

The comment in the comments section was about whether I park-adjusted ERA in creating SIERA. The non-difference was about using park-adjusted BB,SO,GB,FB as inputs.

I didn’t adjust for league average, since I assumed that every year kind of uses the previous year as the run environment for projections, and obviously ERA Estimators are supposed to be the same year. I could check this later.

I actually disagree that RMSE is necessary better than correlation here, but I’m not 100% sure. It’s better if your goal is estimate exact talent level as closely as possible-- so for a straight test of projection systems. But FIP doesn’t do that-- it gives the pitcher his HR rate as his talent, effectively, if used as an ERA projection. Neither does SIERA-- it assumes the pitcher’s strikeout rate is his strikeout rate skill level. So, correlation controls for that, by basically saying, “if each estimator/projection were optimally regressed and re-centered, what would the best guess be?” Does that make sense, or am I missing something about the logic of RMSE vs. correlation. It seems like correlation is the way to do apples to apples, or closer to it, when ERA Estimators are not intended to be projections themselves.

It took me awhile to convince myself, but Matt does have a valid point. When we only care about projections, RMSE is absolutely the right choice, no questions asked. But in this case, if all we want to know which has the strongest relationship, then sure, correlation is fine.

That said, Matt’s article is focused on projections. You would never include projection systems as a measure of a player’s ability. Ability metrics, such as SIERA, are useful for inference. Forecasts are nice for predictions. In baseball, it just so happens that most players don’t change much from year to year, so the inference metrics can be doubled as prediction metrics without worrying too much (of course, inference metrics don’t account for the player being a year older, which would inevitably help for predictions). However, I’m not sure I would agree with the converse here. For that reason, if you are going to include SIERA and Marcel in the same article, you have to be talking about predictions. And thus RMSE is clearly the best choice here.

y=mx+b

RMSE forces m=1

Correlation will find the best-fit for both m and b.

So, if someone forecasts a range of ERA 3.75 to 4.25, a correlation will get us an m greater than 1.  So, even though the forecasting system went out of its way to regress the data toward the mean (and maybe OVERREGRESS incorrectly), allowing the correlation to undo the damage by getting m=2 or something is cheating.

You can’t do that, hence the reason the correlation won’t work.

I had a very long post about this a year or two ago, using Marcel and Pujols as the example.  Hopefully, googling Marcel Pujols forecasting correlation RMSE will find it easily enough.

That was easy.  Post 45:

http://www.insidethebook.com/ee/index.php/site/comments/testing_the_2007_2010_forecasting_systems_official_results/#45

That’s why as a pure projection system test, RMSE works best. However, as a potential projection system derived using ERA Estimators, correlation basically figures out how much to regress (set “m” and “b") to get a projection derived solely from the ERA estimator. It turns out that regressing SIERA about 30% towards the mean does a better job than anything else.

I set the mean of each estimator to be the same for each year, and then re-tested. The order was the same:

SIERA:  1.126 -> 1.108
Marcel: 1.132 -> 1.113
PECOTA: 1.141 -> 1.117
ZiPS:  1.143 -> 1.123
xFIP:  1.148 -> 1.133
FIP:  1.212 -> 1.195
tERA:  1.236 -> 1.214
ERA:  1.387 -> 1.351

Basically, you might as well always subtract about the same amount since they’re all starting from the same point. If I use post-dicting projection systems re-derived from old data, then I’d need to do this for sure, but this way, it seems like tomaeto/tomahto.

Matt: yes, for the non-forecasting systems, you need to do the correlation, to turn them from estimators into forecasters.

FIP for example is not a forecaster, and so, it would make no sense to not change the slope.  SIERA as well.

My original point applies only to forecasting systems, those that had already decided on the slope.

By the way, can you include (K-BB-HBP)/PA in all future testing (and best-fit the slope)?

Tango-- I can but this was in Excel so it’s a little harder to code it right in for this, but I can include it. I did remember to for the SIERA testing this summer, but just forgot here, since I was just trying to do vlookups and match the spelling of the names.