Filter posts by...
Forecasting
Wednesday, February 08, 2012
This was quite a surprising claim from Colin:
These values are on a very different scale, since due to the lack of an intercept the values have to sum to one for the first regression and to three for the second regression, but they’re also very different in a more meaningful sense; recasting the first year to 1 (which is practically already done for us), we get weights of 1/.92/.90.
As you know, Marcel uses 5/4/3 for hitters (meaning 1, .8, .6) and 3/2/1 for pitchers (1, .67, .5). (I think it was 3/2/1… can’t confirm right now.)
I personally use .9994^daysAgo for hitters and .9990^daysAgo for pitchers, which has the effect of being 1, .8, .64, .512 (and so on, each 80% of the previous) for hitters, and 1, .7, .49 (and so on, each 70% of the previous) for pitchers.
Tests from other research makes me think that it should be even more aggressive, so maybe 1, .7, .5 for hitters and 1, .5, .25 for pitchers. But, I haven’t researched that, so, I’ll just leave it there for now.
Colin has gone way to the other side, essentially going with a .9998 or .9999^daysAgo kind of model.
Now, I agree with the framework for his testing, that you should and must include the PA component when establishing the weights. Frankly, this is an important step. When I did it for Marcel, I basically forced everyone in the system to have at least 300 PA, so that I didn’t have to worry about this portion too much (I should have worried a little about this, at least). Indeed, if you give everyone at least 500 PA for each of the three years, this step becomes basically unimportant (no worries at all). That’s because the weighting of each year (the PA of year 1 divided by the PA of years 1 + 2 + 3) will be the same for each wOBA of year 1, 2, and 3.
So, getting back to Colin’s important point: he’s saying that if you introduce the PA weighting component, we see that every year is important. I find this very hard to believe. I mean, it’s an exciting finding if true, and I’d like to see more research on this for sure. My guess at the moment is that there’s a selection bias issue, with guys of limited number of years, or for young guys.
Basically, does Colin’s finding apply across-the-board, or is it really limited to a subset of the population? I’d bet on the latter, and I’d bet that the Marcel 5/4/3 would still hold for players who are regulars. In any case, it’s an exciting prospect to consider.
***
A correction to Colin’s note here:
The third, and perhaps most important, takeaway has to do with regression to the mean. We can add a simplistic version of regression to the mean to our forecasting model by adding a TAv_REG of .260 (the league average) with a PA_REG of 1200. (The PA_REG comes from the Marcels; it’s included here mostly for the purposes of illustration. The regression component in PECOTA is a more rigorous model based on random binomial variance—again, the purpose here is only to illustrate the concepts.
Consider a player with 650 PAs in three straight seasons, or 1950 total PA. Using the Marcel weighting of 1/.8/.6, that comes out to 1560 effective PA— in other words, throwing out 20 percent of a player’s PAs during that time period. That means 56 percent of a player’s forecast comes from his own performance, and 44 percent comes from the regression to the mean component. Using weights of 1/.92/.90 yields 1833 effective PA, throwing out only about six percent. Using the same regression component, that’s 60 percent of a player’s forecast coming from his own production and only 40 percent coming from regression to the mean. (And if you follow from the conclusions above and start using more years to forecast a player as well, even less regression to the mean is necessary.)
There’s a calculation error in there. Marcel uses 5/4/3/2 model, with the 5/4/3 being the weights for years T, T-1, T-2, and the 2 being the weight for regression toward the mean (using 600 PA as the seasonal number). So, if you had say 700 PA in year T, 400 in year T-1, and 500 in year T-2, you get these effective weights:
year T: 700 x 5
year T-1: 400 x 4
year T-2: 500 x 3
regression: 600 x 2
That 600x2 is the same for everyone. Colin’s calculation error is that rather than using 5/4/3, he used 1/.8/.6. The net effect is that he showing a far bigger regression amount than Marcel is actually doing.
Wednesday, February 01, 2012
Brian doesn’t blindly follow his off-the-wall forecast. Good for him.
Sunday, January 29, 2012
I looked at the last 10 firstbaseman (born not later than 1973) to have had at least 10 WAR from age 25-27. These 10 players averaged 13.8 WAR, right in line with Prince’s 14:
WAR Born Player
17.7 1973 Todd Helton
10.9 1973 Mike Sweeney
14.8 1970 Jim Thome
19.4 1968 Jeff Bagwell
18.3 1968 Frank Thomas
17.3 1964 Will Clark
13.2 1964 Rafael Palmeiro
15.2 1963 Fred McGriff
10.9 1963 Mark McGwire
10.1 1963 Cecil Fielder
I got a chuckle at #10 on the list.
Anyway, how did these guys do over the next 9 seasons? I added a column to the above chart called “WAR9”, which is the number of WAR from age 28-36:
WAR9 WAR Born Player
37.2 17.7 1973 Todd Helton
10.6 10.9 1973 Mike Sweeney
40.5 14.8 1970 Jim Thome
50.6 19.4 1968 Jeff Bagwell
33.9 18.3 1968 Frank Thomas
26.0 17.3 1964 Will Clark
41.9 13.2 1964 Rafael Palmeiro
23.1 15.2 1963 Fred McGriff
43.9 10.9 1963 Mark McGwire
4.8 10.1 1963 Cecil Fielder
The average is 31 WAR. If we start a player at 4.8 WAR, and gradually accelerate his aging, we get this kind of aging chart, along with the cost per win (starting at 5MM$ per win, and increasing at 5% each year):
WAR $perW Value
4.8 $5.00 $24.0
4.7 $5.25 $24.7
4.5 $5.51 $24.8
4.2 $5.79 $24.3
3.8 $6.08 $23.1
3.3 $6.38 $21.1
2.7 $6.70 $18.1
2.0 $7.04 $14.1
1.2 $7.39 $8.9
The total comes in at 9 years, 183MM$.
If we take out Cecil Fielder (for whatever reason you want), the other 9 comps average out 34.2 wins, and that would work out to 201MM$.
So, we can create some reasonable scenario where the overpay is some 13MM$ to 33MM$, rather than the 50-100MM$ being discussed.
Friday, January 27, 2012
Anyone going to step up? Anyone? The hard part is collecting all the data, and matching all the players. If someone ELSE does all that hard work, I can step in and do the rest.
The test is pretty simple.
1. Calculate wOBA for every forecast, and for the actual. I’ll do something simple like
numerator = 0.7*BB + 0.9*1B + 1.3*(2B+3B) + 2*HR
denominator = BB+AB
It really doesn’t matter much what you do here. You just need something that focuses on the important stats, and make sure everyone forecasted those stats.
2. Calculate each population mean, by weighting by actual PA (AB+BB). For missing players, either give them the population mean (you HAVE to do this for Marcel, since by definition, Marcel has no missing players), or set the wOBA at 20 points below the rest of the population mean.
3. Recalculate new population mean (where applicable).
4. Baseline each player to a common mean (set to .330, but it doesn’t really matter what you set it to). So, if the pop mean in #3 is .327, and you have a player forecasted for .377, his adjusted forecast is .380.
5. Calculate the difference for every player.
6. Present the average absolute difference, and the RMSE, and in both cases, weighted by the actual PA.
That’s it, that’s the basis.
Then you can do fun stuff, like splitting by career performances. Guys with 1500 or more PA in the last 3 years, guys with fewer than 250 PA, guys who had a .380+ wOBA in the last three years, shortstops, etc, etc. Look for whatever attribute of a player you want. And compare the systems, and look for bias.
Bueller?
Saturday, January 21, 2012
This time, it’s Clay’s turn.
I’m playing with my son right now, so I can’t comment much. I see Rangers have the lowest runs allowed in the AL, which would seem hard to do in that park. And the spread in wins seems wide. Can someone calculate the standard deviation of wins? A good forecast should have one SD = 9 or so.
Wednesday, January 11, 2012
That’s the implication here. But, seeing that the quality of pitchers in the starters group FAR exceeds that of the relievers group, it wouldn’t necessarily be a starter/relief thing, but a good/bad pitcher thing.
I’ve mentioned in the past that it’s more likely that good players don’t age as fast or peak as early as bad players. That’s part of what makes them good. Plenty of players peak at age 21, and chances are, they weren’t that good.
Friday, January 06, 2012
Dave has a good piece on Adam-Jones like players.
Thursday, January 05, 2012
Matt shows this:
Estimator(N=1,576 pitchers) RMSE of Statistic with Next Year’s ERA(2006-2011)
SIERA 1.126
Marcel 1.132
PECOTA 1.141
ZiPS 1.143
xFIP 1.148
FIP 1.212
tERA 1.236
ERA 1.387
First of all, no need to go to three decimal places. We show ERA as two points, so why bother showing it to three decimal places? As far as I’m concerned, there’s virtually no difference among the top five.
Secondly, I can’t tell if the future ERA is park-adjusted or not. It MUST be unadjusted. NO ONE is trying to estimate a pitcher’s park-neutral ERA in terms of testing. The only test is how he actually did. So, we don’t adjust for park and strength of schedule and innings per start.
(MGL for example only cares about park-neutral. And that’s fine. But then, we can’t test his results. SIERA is park-neutral I think, but FIP is not. All the forecasting systems in fact are park-specific. You can’t turn everything to park-neutral first.)
You COULD make the case that we should throw out any unexpected starter-relief switches, for reasons we’ve learned about over the years. But, we need to be careful here, as we may end up with a selection bias.
In the comments, Matt notes that it was park-adjusted. Again, I completely disagree here. The test is against actual performance, not adjusted performance. He notes it didn’t make a difference. Well, given that the test is slanted toward SIERA, and SIERA is Matt’s baby, then, I’d REALLY like to see the results the right way.
Now, Matt may decide to introduce a park-specific SIERA, so that we can all make the apples-to-apples comparison. Until then, SIERA will simply have to have its hand tied behind its back.
Thirdly, for the RMSE test, you MUST calibrate it so the league average for the forecast equals the league average for the actuals. It should be clear that if you treat the forecasting system as its own universe, it’s irrelevant if the expected ERA was set to 3.9 or 4.3 and the actual ended up at 3.7 or 4.8 or whatever. I’m not sure if Matt handled this.
As we know, RMSE, not correlation, is the correct test.
Having said all that: great job to Matt!
Thursday, December 01, 2011
Fangraphs has them up.
Tuesday, November 29, 2011
I ran four competitions, three unofficial, and one official. I’ll run them all down. I’m going to list the results of all the pro forecasters who finished ahead of Marcel. For those that finished below Marcel, I will list them in alphabetical order.
Read More
Monday, November 28, 2011
I have several posts on Bill James’ site suggesting giving Cliff Lee a 20% chance at 300 wins is quite optimistic.
It’s throughout the comments of this article (which is free, as are all the comments, including Bill’s). It’s a decent read.
For posterity, all my posts are being reproduced below.
Read More
Dave looks at Gonzalez and Volquez, two pitchers who seemingly have put up similar stats over the last 4 seasons, but with a different “path”. He shows what happens when you re-weight.
This is one place where it would be interesting to know more about the fundamental changes, if any, of pitchers.
Tuesday, November 22, 2011
KJOK was nice enough to send me a Marcel file that includes the 2009 forecasts, the 2009 actuals, and if the player played for the same team in 2008 and 2009.
I took his data, and calculated the wOBA for each player (his forecast, and his actual). I found the weighted error as the difference between these two figures, multiplied by his actual PA.
I limited the forecasts to only those players with a reliability of at least .50, and who, naturally, played in 2009. This gave me a total of 382 hitters, with 151,897 PA.
I then simply split up the 382 players into whether they played for the same team or not. For those that played on the same team, the average error was .025. For those that switched teams, the average error was .027.
Since Marcel does not make a park adjustment, that could be a source of error. Then again, it could simply be that the change in context simply forces an error. In order to understand if park adjustment is the reason or not, we need to compare to a forecasting system that does an explicit park adjustment. (It may be that the team switchers simply are harder to forecast because there was a reason they switched teams to begin with.)
If someone has forecasts for 2009 that were made with a park adjustment, then please supply me with such a file. It MUST contain at least the following information:
bdbID, AB, H, 2B, 3B, HR, BB, HBP
If you don’t have at least exactly that, don’t send me anything!
I will then have something to compare against.
Bill James:
I used to do projections for players just for fun. After we started the Handbook (about 1990) John said “Why don’t we publish those projections.”
“John,” I said, “those projections don’t have any scientific validity whatsoever. I’m just messing around with formulas, playing around with them. I can’t publish that.”
“We won’t say they have any validity,” John said. “We’ll just say we do these projections, and you can take them for what they’re worth.” So we started printed them, and people liked them, so we still do it.
A perfect and honest response: Just for fun, no scientific validity, take them for what they’re worth.
Beautiful.
Monday, November 21, 2011
Clay notes how embarrassing, and otherwise confusing, decades old code looks like. All I can say to that is: guilty! When you don’t follow standards, things look so messy in a few years, that you not only try to avoid looking at the code, sometimes you just end up re-writing it entirely.
That’s a lesson for you kids: get it right the first time, by taking your time. That’s why what I’ve done for the last several years is include a “readme” file in every new folder I create. It’s basically what we call a “run book”, so that if someone comes in cold, you know exactly what needs to be done, if you start from scratch. It’s tremendously helpful.
Anyway, that’s not really the reason I linked to his article. What caught my eye is this:
I’ve also been validating projection systems from the 2011 season. While I’m pleased with how my system (which ran with some of Nate Silver’s ideas on PECOTA, threw out some of them, replaced them with some of my own tools and approaches, resulting in a chimeric Sildavenverport monster) graded, and I was also pretty shocked at just how little difference even the most complex systems made when compared to an ordinary three-year average.
This is exactly why I created Marcel some eight years ago:
http://www.tangotiger.net/archives/stud0346.shtml
And why I thought so little of forecasting systems that I published the code so you can create it yourself. And I thought so little of systems precisely because I spent countless hours trying to beat myself each time. I’d come up with the basics, then think of different parameters, and trying to combine them in different ways, to improve my system. And each time, the gains would be so negligible, that the gain was hardly worth the time.
Even things like park factors, which I presumed would make a huge difference, hardly made a dent. And when it came time for pure rookies (guys who never played in MLB), systems who were designed with extreme intelligence on the matter (Rally, MGL, ZiPS, PECOTA) barely were any better than if we just presumed the players were ALL THE SAME (while Marcel uses league average out of convenience, it’s better to just use the first-year average, or about a wOBA of 15 points under league average). The rookies thing, the MLEs, is ripe for selection bias that makes it basically impossible for those systems to beat the most basic system. Not to mention it makes an enormous difference if the rookie is going to be a reliever or starting pitcher.
It’s not like I just took some position on the matter, and am defending it. I took this position because this is where the path has led me after countless hours spent studying this matter in as many ways as I can. And it’s been re-affirmed when testing systems of other people smarter than me and who spent more time than I have, only for those people to be perhaps one step above Marcel. If you need a visual here, we all started on Canal Street, and while Marcel is at Penn Station, those systems are on 35th street, and Times Square is just outside our reach.
Any forecaster honest with himself, and his readers, will attest to this.
Thursday, November 17, 2011
To me, the story here is that having a single person setting a line is not a good thing.
Now that the season is over and we are into awards season, it’s time to announce a winner. By a landslide, the most prescient prognosticator this year was Matthew Kenerly, who ran down Rex Babiera in the home stretch by choosing the correct side of the line on 39 of 50 players. No one else had more than 37 correct, so Matthew showed himself to be head-and-shoulders above the crowd and has our permission to proclaim himself the wisest of all BP readers, a title I’m sure will earn him due deference during comments section discussions throughout the coming year. Less importantly, Matthew has won himself a free copy of Baseball Prospectus 2012 with as many author signatures as I can manage to round up this spring. Well done, Matthew.
As is often the case, the Wisdom of Crowds also performed admirably, with the majority of entries making the correct choice 36 times. Pitchers and hitters were equally difficult to predict, with the groupthink entry finding the right answer on 60 percent of both groups.
If Ken set the perfect line, we should have had 50 percent getting the right answer. What Ken has proven here is that the group can set the better line than Ken. And, I’m going to extend that to say that the group can set the better line than an individual.
This has always been the case for as long as I’ve done these kinds of studies, and seen these kinds of studies, so big thanks to Ken for showing how difficult it truly is for any one person to set a good line.
For example, I did this a decade ago. Focus on the individual black lines, and the green line. The green line is the average of all the black lines. This is why the group is always better than the individual, unless the individual is really really good (all the red lines below), in which case, the individual will be as good as the group (the green is in the middle of the reds).
Tuesday, November 08, 2011
Xeifrank has a great post on his blog. He converts all the game-by-game odds of Vegas into a win probability for each team (and tracked it by starting pitcher). Then, he simply compared how many games the team actually won to how many Vegas expected that team to win, with that pitcher.
For Felix Hernandez for example, Vegas had no surprise: the Mariners won exactly as many games as Vegas expected. The DBacks, with Kennedy on the mound, won alot more. Tigers/Verlander as well.
Basically, either it took time for Vegas to catch up to the change in the estimate in the true talent level of Kennedy, Verlander, etc, or, there was alot of good luck in the wins of Kennedy and Verlanders, etc.
How to tell? Well, compare the differences between actual and expected, and see if that matches to what you’d expect from random. If it’s an exact match, then we know that the differences were all luck. (i.e., expect Vegas to not change their estimate in true talent level for Kennedy, Verlander et al). If on the other hand the differences are wider than expected, then Vegas was a little slow in adjusting for the estimate in change in talent levels.
blogspot is blocked at the office, so I will rely on a Straight Arrow reader to help us out.
Friday, November 04, 2011
Pretty much everything that’s been said in the 2011 pre-season about Carl Crawford can apply to Jose Reyes today.
Here’s one take on it.
Friday, October 28, 2011
Ryan:
Monday, October 24, 2011
Brian argues:
I’m in favor of good character and the will to win as much as the next guy. But to the extent these qualities influence play on the field, the numbers will capture their effect. So the next time your buddy says stats don’t measure the intangibles, you can say, ‘Sure they do. And besides, where have these magical qualities been hiding all season? And why haven’t they shown up on the field until now anyway?’
I think there are two points here:
1. There’s no question that if you have some great intangible quality (you work at the children’s hospital in your off days), but that quality has no direct or indirect effect on your performance or your teammates’ performance, then it’s not relevant.
2. You have your great “work with kids” intangible quality, and it DOES have a direct or indirect effect, then we WILL see the result in a better performance by you and more wins by your team. You have a great work ethic, you keep your teammates loose, you have whatever other intangible quality that is highly prized: the impact will be felt, eventually, on the scoreboard.
***
What I think is really being discussed though is that the player with the great intangibles will go on to have an even better career than the guy with not, if they both have the same stats to that point.
For example, Derek Jeter and Manny Ramirez, through the 20s, might be considered to have around the same win impact to their teams. Jeter may have had less innate skill, but more intangibles, while Manny may have had more innate skill, but less intangibles (or even “negative” intangibles). The question then becomes: who will have more win impact in the future?
Now, it’s possible that the intangible-heavy player will age better, and that the intangible-light player will age terribly. This is, basically, what the “anti-spreadsheet” crowd is really arguing.
And, to a certain extent, I am with that crowd. Take for example, everyone’s favorite replacement-level player: Willie Bloomquist. A player who is barely able to make the 25-man roster of any team he’s been on is a prime candidate for being out of MLB the next year. Once a player hits the age of 30, he loses about 0.5 wins of value each year. A player is, after all, human, and not a machine. His body just can’t handle the rigors of time.
Wille Ballgame will turn 34 years old next month. His career hitting is at 80% of the league average. He’s held steady at 80% of the league average over the last 2 years, last 5 years, and his career. It’s as if he’s not aging. His hitting does not get better, nor does it get worse. His fielding may be similarly unaffected. Willie Ballgame’s intangibles may simply be offsetting father time.
Maybe there’s other guys like him. Craig Counsell maybe? Counsell has the same offensive non-deterioration as Willie Ballgame (though perhaps even his intangibles can’t counteract father time after age 40). Counsell was a far better fielder than Willie, and has survived in MLB on basically his fielding counteracting father time.
It’s possible that I’m just cherry picking players, building a narrative. Gary Sheffield for example would probably be a prime candidate for being intangible-light, but he was probably as good in his early to mid 30s as he was at any point in his 20s.
In any case, I think this is really want people are talking about separating the intangibles from the physical: the physical will deteriorate at a very fast clip, while intangibles are like diamonds.
from Amazon
Recent comments
Older comments
Page 1 of 320 pages 1 2 3 > Last »Complete Archive – By Category
Complete Archive – By Date