THE BOOK--Playing The Percentages In Baseball

Monday, February 08, 2010

Evaluating the 2009 forecasts - Chone/ZiPS + Fantistics win

Professor Jared and his merry band of high school students take on the task.  Those ambitious fellows created the Steamer forecasts, and they submitted it as part of my Forecasters Challenge last year.  They finished 6 out of 22 in the official competition, and number 2 out of 22 in the rules I’ll be using for the 2010 competition.  And they did this while submitting an abbreviated list of players (which I supplemented with the Marcel draft list in the late rounds).  Yes, we should all be impressed. 

Anyway, let’s see what their analysis reveals. They compared these systems: Steamer, Marcel, PECOTA, Chone, ZiPS, Sporting News, Fantistics.

Missing data - 475 hitters had 50 or more PA in 2009.  465 of these hitters had projections from each of the big 3 (chone, pecota and zips).  438 were projected by Marcel.  We looked at these 438 hitters.  Projection systems that projected fewer players (Steamer Projections and Sporting News were the main guilty parties) were given the Marcel projection for that player.  This allowed for a comparison of all 438 hitters across systems.  Sporting News and Steamer only projected about 270 players each.  Systems could beat the monkey so long as the projetions they actually made were better than Marcel.

A technical note: Marcel’s official forecast for anyone not in the downloaded file is:

FAQ: “But, what about a player who’s never played MLB? Where’s his forecast?” That’s simple. His forecast is the league mean over 200 PA, 60 IP (starter) or 25 IP (reliever). If you want to know what the league mean is, just take the average of anyone forecast with a reliability of 0.00. So, Marcel’s official forecast for anyone coming over from Japan is that.

So, to be fair to The Big 3, the 27 missing players should be included with a forecast exactly as I said it should be.  It makes no sense for me to include it explicitly in the download file, if it’s just going to be a line of data that repeats for all players.  But, this is what should be done, because that’s what I said it is.  This is using Fantasy points:

RMSE* System
2.41 Avg Projection
2.43 Fantistics

2.49 Sporting News

2.52 Marcel
2.53 Steamer
2.55 ZiPS
2.56 Chone

2.65 PECOTA
2.67 2008

Holy moley.  First of all, Marcel had a great year.  Secondly, PECOTA did so bad, that it was as accurate as simply taking the previous season’s stats and running with those.  Don’t like RMSE?  How about correlation:

R with actual System
0.729 Avg Projection
0.723 Fantistics

0.707 Sporting News

0.697 Marcel
0.696 Steamer

0.688 ZiPS
0.687 Chone

0.657 PECOTA
0.653 2008

Again here, Marcel did very well, while PECOTA got trounced.  This is pretty shocking actually.  I don’t know what happened to PECOTA, but I’d love to see Baseball Prospectus explain it.  And who the heck is Fantistics anyway?

Anyway, let’s go on:

Also worth noting, each of projection systems has a smaller standard deviation across their projections than the standard deviation of actual results from 2008 or 2009.  This is as it should be.  The projection systems are trying to forecast true talent whereas the variance in actual results is a combination of the variance in true talent and the variance in luck.

Nothing novel there, but it should be noted because lots of people who are new to this aren’t aware.

Continuing:

And, if you want the best linear equation of these systems for projecting actual 2009 SGPs:
Actual = 0.527*Fantistics + 0.409*Chone + 0.243*SportingNews – 0.761 (R2 = 0.547)
While Sporting News projected SGP’s better than Chone, Chone added more information to Fantistcs because Chone was the most unique system while Sporting News was the least unique system.

Very interesting.

And this is using OPS:

RMSE* System
1.55 Chone
1.57 ZiPS
1.57 Avg Projection

1.62 Marcel
1.63 Fantistics
1.65 Sporting News
1.66 Steamer
1.66 PECOTA

1.84 2008

Chone and ZiPS lead the way.  Marcel is where it should be, whil PECOTA once again brings up the rear (just not so obvious this time).  And if you prefer r:

R with actual System
0.638 Chone
0.624 Avg Projection
0.623 ZiPS

0.590 Marcel
0.583 Fantistics

0.568 Sporting News
0.567 Steamer
0.564 PECOTA

0.401 2008

The usual story.  Chone does great, Marcel does average, PECOTA is in last place.

And, if you want the best equation to project 2009 OPS:
ActualOPS = 0.716*Chone + 0.199*ZiPS + 0.081 (R2 = 0.414)
Although this equation doesn’t do much better than simply using Chone.

So, if Fantistics was only in the middle of the pack in projecting OPS, how did it dominate SGP’s?  Ok, so Chone and ZiPS aren’t really trying to project playing time and, despite their excellence in projecting hitter quality (as evidenced by OPS) don’t do well here.  Pecota doesn’t try to project playing time in their weighted mean forecasts but does in their depth charts (used by Steamer).  The community forecasts that Marcel used do reasonably well, but not as well as the fantasy basebal gurus in projecting playing time.  Limiting this to the systems that try to project playing time (and using their proper names this time) we have:

R with actual System
0.721 Fantistics

0.694 Sporting News

0.666 Pecota Depth Charts for Fantasy
0.657 Community Forecasts

It looks to me like we found our secret recipe: Chone/ZiPS forecasts for rate, with Fantistics for playing time.  What I’d like for the professor and his kids to do is to run their study that uses Chone/ZiPS for rates and Fantistics for playing time.