THE BOOK--Playing The Percentages In Baseball

Good idea to look at the Chone/ZiPS + Fantistics super system.  I think that’s what I’d use for a 5x5 league this year.

To be fair we should also try Pecota with their depth chart playing time instead of their weighted mean playing time.  While they didn’t do that well, projecting hitter quality it’s really their SGP’s which were especially brutal and I think was in large part due to the playing time projections.

I think for PECOTA, since they actually publish forecasts that are explicitly intended to be used as-is, then what you did was totally fair game.

If you wanted to include an additional PECOTA data point, that would also be fair.

***

ZiPS and Chone explicitly said they don’t really forecast playing time.  Indeed, ZiPS, in the Forecaster’s Challenge, used the same playing time forecasts that Marcel and MGL used: community.  And Chone will be using some form of community playing time forecasts as well.

Seeing the results, it certainly seems that the #1 thing to get right is the playing time forecasts.

So, it will be interesting to see how the Fangraphs Community does in 2010.

(sarcasm) If we exclude the Matt Wieters for MVP forecast, would that help PECOTA? (/sarcasm)

Did PECOTA have any sytematic bias?  Or were they all over the map?

Heh, actually Wieters was left out completely b/c this only included guys with a Marcel reliability > 0.

It did occur to me that we could give guys without Marcel projections the default projection and go from there.  This might have sunk the systems that didn’t project very many guys however, despite the fact they they projected enough players for most fantasy drafts.  Someone could do just fine in a fantasy league without drafting any rookies. 

For pitchers I’m thinking we should project anyone who is projected by marcel (reliability > 0) PLUS anyone who had either 10 game starts or 10 saves.  This would “punish” systems that failed to project players of fantasy significance while not being overly harsh.

What do you think?

Jared: excellent point for the pure rookies.  Yes, Marcel did NOT include in its draft list guys like Elvis Andrus, and neither did Steamer.  And those two systems did GREAT in the Forecaster’s Challenge.

So, yeah, I agree, and it’s a great point.

That said, it would still be a bit more fair to also include the pure rookies, just to head off the complaints, as minor as they’ll end up being.

It’s an interesting point to select players, after the fact, of ONLY the guys who did do well.  It would be totally wrong of course to include in a sample only players who you’ve seen the results on.  It would give those systems a leg-up.

Then again, you are weighting the forecasts based on their actual playing time.

I’d say report it as two separate things: one without the pure-rookies, and one that includes pure-rookies.

The best way to describe my playing time forecasts is an estimation based on recent playing time data of how much the player is capable of playing.  I do not specify that such playing time will be in MLB.  I think it’s reasonable that Peter Bourjos gets 500 PA or so this year, but 90% of them will be in Salt Lake City, and maybe he’ll get a September callup to the big A.

Thank you J Cross and students for doing this analysis.  I see where the strengths of my system lie, and that is player ability.  Playing time projections take a human touch, and while I attempt to do that (unpublished) for my team W-L projections, I don’t think they are publication quality.  It’s tough for one person to keep up with 30 teams that way, a team approach to depth charts seems like the best way to go.

Jared, Let me know if you guys want an updated spreadsheet anytime to combine with your PT projections.  I think a good collaberation could be found here, and if you guys are willing I could create a special set of webpages with your PT and my rates together.  And if you annoint closers for every team, maybe I can finally have saves on my site as well.

email for me is rally monkey (numeral five) at comcast dot net.

Thank you very much! This is exactly what I’ve been looking for.

Rally, we haven’t tried to make playing time projection yet although I think we will.  I’d definitely be game for combining them with Chone projections once we do.  They might not be any good though.

Last year, our spreadsheet predicted save opps for each team and then Peter simply spread those save opps among the team’s relievers based on his baseball expertise and we based sv% on ERA.  Not sure that our “system” will have done so well but we’ll see.

After this year, we’ll be able to see if there are meaningful biases in the fangraphs fan forecasts.  I’m guessing that they predict relatively too many at bats to older players.  Every system other than fantistics had a statistically significant age bias.  It seems like “fans” community, the prospectus depth chart guru and the sporting news guru) project too many at bats for older players while Zips, Pecota and Chone projected too many at bats for younger players.

When I projected saves last year, I found team save opps by dividing projected team wins by two (a surprisingly accurate shortcut), and then handed out saves based on ERA--i.e., a closer with a 2.50 ERA got 85% of the teams total save opps, for instance. I chose the closer for each team, and the remaining slots were sorted based on ERA.

An 80 win team with 40 save opps, with the following relievers and their save %’s (picked out of thin air but lower ERA = higher %):

2.50, 85% ... 85% of 40 save opps is 34 saves
3.00, 80% ... 80% of 6 remaining save opps is 4.8 saves
3.50, 75% ... 75% of 1.2 remaining save opps is 1 save

I have no idea if this method is accurate (especially the save percentages based on ERA), but it passes the eye test.

Just to make it clear, those percentages are projected save opps / team save opps (or remaining save opps). That method can’t predict BS or individual save percentage.

Jared, I would appreciate if you would be able to evaluate my Oliver projections. Click on my name for my email address and I can send you a data file.

Update:

Oliver also beats Marcel

R w/ actual:

R with actual System
0.638 Chone
0.624 Avg Projection
0.623 ZiPS
0.607 Oliver
0.590 Marcel
0.583 Fantistics
0.568 Sporting News
0.567 Steamer
0.564 PECOTA

0.401 2008

J. Cross, can you clarify how you’re figuring RMSE for OPS?

Bumping.

Because I literally have no idea what an RMSE of 1.66 means in the context of OPS. What are the units on that?

Colin, those numbers look funny it’s because it’s the RMSE of OPS weighted by PA.  If you divide it by the square root of the mean (harmonic mean?) PA of the population you should get a number that looks like what you might expect for the RMSE for OPS (but with each error weighted by the number of PA).  I probably should have done this or just given RMSE for the OPS not weighted by PA.

Looks like the mean PA was 384 and here’s what I probably should have given:

RMSE for OPS (unweighted)

‘08: 0.109
chone: 0.093
steamer: 0.097
marcel: 0.096
pecota: 0.098
ZiPS: 0.094
sn: 0.097
fantistics: 0.098

If I take the numbers in my write up and divide by sqrt(384) I should get numbers slightly lower than these because the projections were more accurate for guys with more PA.

make more sense?

btw, I envy you getting to play around with the inner workings of pecota.  That actually sounds like a lot of fun.