Buy The Book
I have several posts on Bill James’ site suggesting giving Cliff Lee a 20% chance at 300 wins is quite optimistic.
It’s throughout the comments of this article (which is free, as are all the comments, including Bill’s). It’s a decent read.
For posterity, all my posts are being reproduced below.
Dave looks at Gonzalez and Volquez, two pitchers who seemingly have put up similar stats over the last 4 seasons, but with a different “path”. He shows what happens when you re-weight.
This is one place where it would be interesting to know more about the fundamental changes, if any, of pitchers.
KJOK was nice enough to send me a Marcel file that includes the 2009 forecasts, the 2009 actuals, and if the player played for the same team in 2008 and 2009.
I took his data, and calculated the wOBA for each player (his forecast, and his actual). I found the weighted error as the difference between these two figures, multiplied by his actual PA.
I limited the forecasts to only those players with a reliability of at least .50, and who, naturally, played in 2009. This gave me a total of 382 hitters, with 151,897 PA.
I then simply split up the 382 players into whether they played for the same team or not. For those that played on the same team, the average error was .025. For those that switched teams, the average error was .027.
Since Marcel does not make a park adjustment, that could be a source of error. Then again, it could simply be that the change in context simply forces an error. In order to understand if park adjustment is the reason or not, we need to compare to a forecasting system that does an explicit park adjustment. (It may be that the team switchers simply are harder to forecast because there was a reason they switched teams to begin with.)
If someone has forecasts for 2009 that were made with a park adjustment, then please supply me with such a file. It MUST contain at least the following information:
bdbID, AB, H, 2B, 3B, HR, BB, HBP
If you don’t have at least exactly that, don’t send me anything!
I will then have something to compare against.
Bill James:
I used to do projections for players just for fun. After we started the Handbook (about 1990) John said “Why don’t we publish those projections.”
“John,” I said, “those projections don’t have any scientific validity whatsoever. I’m just messing around with formulas, playing around with them. I can’t publish that.”
“We won’t say they have any validity,” John said. “We’ll just say we do these projections, and you can take them for what they’re worth.” So we started printed them, and people liked them, so we still do it.
A perfect and honest response: Just for fun, no scientific validity, take them for what they’re worth.
Beautiful.
Clay notes how embarrassing, and otherwise confusing, decades old code looks like. All I can say to that is: guilty! When you don’t follow standards, things look so messy in a few years, that you not only try to avoid looking at the code, sometimes you just end up re-writing it entirely.
That’s a lesson for you kids: get it right the first time, by taking your time. That’s why what I’ve done for the last several years is include a “readme” file in every new folder I create. It’s basically what we call a “run book”, so that if someone comes in cold, you know exactly what needs to be done, if you start from scratch. It’s tremendously helpful.
Anyway, that’s not really the reason I linked to his article. What caught my eye is this:
I’ve also been validating projection systems from the 2011 season. While I’m pleased with how my system (which ran with some of Nate Silver’s ideas on PECOTA, threw out some of them, replaced them with some of my own tools and approaches, resulting in a chimeric Sildavenverport monster) graded, and I was also pretty shocked at just how little difference even the most complex systems made when compared to an ordinary three-year average.
This is exactly why I created Marcel some eight years ago:
http://www.tangotiger.net/archives/stud0346.shtml
And why I thought so little of forecasting systems that I published the code so you can create it yourself. And I thought so little of systems precisely because I spent countless hours trying to beat myself each time. I’d come up with the basics, then think of different parameters, and trying to combine them in different ways, to improve my system. And each time, the gains would be so negligible, that the gain was hardly worth the time.
Even things like park factors, which I presumed would make a huge difference, hardly made a dent. And when it came time for pure rookies (guys who never played in MLB), systems who were designed with extreme intelligence on the matter (Rally, MGL, ZiPS, PECOTA) barely were any better than if we just presumed the players were ALL THE SAME (while Marcel uses league average out of convenience, it’s better to just use the first-year average, or about a wOBA of 15 points under league average). The rookies thing, the MLEs, is ripe for selection bias that makes it basically impossible for those systems to beat the most basic system. Not to mention it makes an enormous difference if the rookie is going to be a reliever or starting pitcher.
It’s not like I just took some position on the matter, and am defending it. I took this position because this is where the path has led me after countless hours spent studying this matter in as many ways as I can. And it’s been re-affirmed when testing systems of other people smarter than me and who spent more time than I have, only for those people to be perhaps one step above Marcel. If you need a visual here, we all started on Canal Street, and while Marcel is at Penn Station, those systems are on 35th street, and Times Square is just outside our reach.
Any forecaster honest with himself, and his readers, will attest to this.
To me, the story here is that having a single person setting a line is not a good thing.
Now that the season is over and we are into awards season, it’s time to announce a winner. By a landslide, the most prescient prognosticator this year was Matthew Kenerly, who ran down Rex Babiera in the home stretch by choosing the correct side of the line on 39 of 50 players. No one else had more than 37 correct, so Matthew showed himself to be head-and-shoulders above the crowd and has our permission to proclaim himself the wisest of all BP readers, a title I’m sure will earn him due deference during comments section discussions throughout the coming year. Less importantly, Matthew has won himself a free copy of Baseball Prospectus 2012 with as many author signatures as I can manage to round up this spring. Well done, Matthew.
As is often the case, the Wisdom of Crowds also performed admirably, with the majority of entries making the correct choice 36 times. Pitchers and hitters were equally difficult to predict, with the groupthink entry finding the right answer on 60 percent of both groups.
If Ken set the perfect line, we should have had 50 percent getting the right answer. What Ken has proven here is that the group can set the better line than Ken. And, I’m going to extend that to say that the group can set the better line than an individual.
This has always been the case for as long as I’ve done these kinds of studies, and seen these kinds of studies, so big thanks to Ken for showing how difficult it truly is for any one person to set a good line.
For example, I did this a decade ago. Focus on the individual black lines, and the green line. The green line is the average of all the black lines. This is why the group is always better than the individual, unless the individual is really really good (all the red lines below), in which case, the individual will be as good as the group (the green is in the middle of the reds).
Xeifrank has a great post on his blog. He converts all the game-by-game odds of Vegas into a win probability for each team (and tracked it by starting pitcher). Then, he simply compared how many games the team actually won to how many Vegas expected that team to win, with that pitcher.
For Felix Hernandez for example, Vegas had no surprise: the Mariners won exactly as many games as Vegas expected. The DBacks, with Kennedy on the mound, won alot more. Tigers/Verlander as well.
Basically, either it took time for Vegas to catch up to the change in the estimate in the true talent level of Kennedy, Verlander, etc, or, there was alot of good luck in the wins of Kennedy and Verlanders, etc.
How to tell? Well, compare the differences between actual and expected, and see if that matches to what you’d expect from random. If it’s an exact match, then we know that the differences were all luck. (i.e., expect Vegas to not change their estimate in true talent level for Kennedy, Verlander et al). If on the other hand the differences are wider than expected, then Vegas was a little slow in adjusting for the estimate in change in talent levels.
blogspot is blocked at the office, so I will rely on a Straight Arrow reader to help us out.
Pretty much everything that’s been said in the 2011 pre-season about Carl Crawford can apply to Jose Reyes today.
Here’s one take on it.
Brian argues:
I’m in favor of good character and the will to win as much as the next guy. But to the extent these qualities influence play on the field, the numbers will capture their effect. So the next time your buddy says stats don’t measure the intangibles, you can say, ‘Sure they do. And besides, where have these magical qualities been hiding all season? And why haven’t they shown up on the field until now anyway?’
I think there are two points here:
1. There’s no question that if you have some great intangible quality (you work at the children’s hospital in your off days), but that quality has no direct or indirect effect on your performance or your teammates’ performance, then it’s not relevant.
2. You have your great “work with kids” intangible quality, and it DOES have a direct or indirect effect, then we WILL see the result in a better performance by you and more wins by your team. You have a great work ethic, you keep your teammates loose, you have whatever other intangible quality that is highly prized: the impact will be felt, eventually, on the scoreboard.
***
What I think is really being discussed though is that the player with the great intangibles will go on to have an even better career than the guy with not, if they both have the same stats to that point.
For example, Derek Jeter and Manny Ramirez, through the 20s, might be considered to have around the same win impact to their teams. Jeter may have had less innate skill, but more intangibles, while Manny may have had more innate skill, but less intangibles (or even “negative” intangibles). The question then becomes: who will have more win impact in the future?
Now, it’s possible that the intangible-heavy player will age better, and that the intangible-light player will age terribly. This is, basically, what the “anti-spreadsheet” crowd is really arguing.
And, to a certain extent, I am with that crowd. Take for example, everyone’s favorite replacement-level player: Willie Bloomquist. A player who is barely able to make the 25-man roster of any team he’s been on is a prime candidate for being out of MLB the next year. Once a player hits the age of 30, he loses about 0.5 wins of value each year. A player is, after all, human, and not a machine. His body just can’t handle the rigors of time.
Wille Ballgame will turn 34 years old next month. His career hitting is at 80% of the league average. He’s held steady at 80% of the league average over the last 2 years, last 5 years, and his career. It’s as if he’s not aging. His hitting does not get better, nor does it get worse. His fielding may be similarly unaffected. Willie Ballgame’s intangibles may simply be offsetting father time.
Maybe there’s other guys like him. Craig Counsell maybe? Counsell has the same offensive non-deterioration as Willie Ballgame (though perhaps even his intangibles can’t counteract father time after age 40). Counsell was a far better fielder than Willie, and has survived in MLB on basically his fielding counteracting father time.
It’s possible that I’m just cherry picking players, building a narrative. Gary Sheffield for example would probably be a prime candidate for being intangible-light, but he was probably as good in his early to mid 30s as he was at any point in his 20s.
In any case, I think this is really want people are talking about separating the intangibles from the physical: the physical will deteriorate at a very fast clip, while intangibles are like diamonds.
This was a research piece from MGL from 8 years ago. I had no recollection of it. I would bet that MGL has also completely forgotten about it as well.
It’s great stuff, and it’s ripe for some aspiring saberist to update, and perhaps handle the selection bias.
The thread where I introduced Marcel has been translated into Romanian! Thank you to Alexader.
This BPro reader, Juris, says not so:
What caught my eye on this is that you now have the Tigers listed as 100% likely to win their division. But the Magic Number as of this morning is 6 versus the Indians and 5 versus the WhiteSox. Neither of those teams is mathematically eliminated from winning the division, though the probability of either one of them winning is very very small. I would therefore list the Tigers’ chances as 99.9%.
However, Cool Standings ALSO has the same 100%, whether you use the “smart” or “dumb” options (dumb implying every team is a true talent .500 team). So, what gives?
Well, it’s easier to see with Clay’s report here, where he gives the Tigers a 99.99985% chance of making the playoffs.
It’s basically a rounding issue. Clay runs one million sims, and he rounds not at all. That is, out of one million sims, the Tigers make the playoffs 999,998.5 (the fraction presumably because there was a tie that has to go to a one-game “playoff”, but is, technically, not a playoff).
I think it’s probably better, in these cases, that if you need to round, to round away from the extremes (100% and 0%). At the very least, at least cap at 99.9% or 0.1%, unless the sim truly showed 100% and 0%.
Good job Juris! A nit pick close to my heart.
Xeifrank comes in with the great data here. It is only 24 games, so, it’s just something to get people started.
Anyway, 10 games were in reality under, scoring a total of 1 to 5 runs. 14 games were over, scoring 7 to 14 runs.
Indeed, since all the total runs forecasted were all close to 6 runs, and NONE of the games actually scored 6 runs, Vegas would have to drastically alter the odds in order to get an equal number of over and under.
Indeed, if the over/under line was set 1.4 runs higher than what they actually set, it’s at that point that you would get 9 overs and 15 unders, and increasing the line by 1.1 runs gets you 11 overs and 13 unders, which, together, is a mirror to what happened. So, splitting the difference (1.25 runs higher line), and you’d think they should set the line higher by 0.625.
The average total runs predicted was 5.80, while the actual runs scored was 6.42, meaning it was 0.621 runs too low. (Love it when things come together like that.)
Anyway, just a little data point here to start the ball rolling for further research and into past years.
RJ has the data here. Specifically, his 23 teams have a 1st half win% of .399. In the 2nd half, they have a win% of .549. Overall, that makes them .474. That fact alone is enough for us to expect them to post a higher win% in the following year. The question on the table though is if .399/.549 will lead to a higher win% than say .474/.474. Unfortunately, RJ doesn’t give us a control group here. We can estimate it quickly by regressing to the mean by adding 69 games of .500 ball. So, .474 in one year would imply .482 in the next year.
How did they actually do? Their gain year to year was .002, to .476. Indeed, it’s a tiny bit worse than expected (though easily within the bounds of random variation).
If we can’t see the effect in the most extreme cases, then chances are, using this as an indicator is not going to be terribly helpful. You are far better off looking at individual players, rather than looking at change in team wins from the first half to the second half.
Here’s the data I used from his chart:
Some interesting stuff from Brian, as he walks you through some of the pitfalls.
Note that the same “last year drop” happens in baseball too. The reason is that this is a sampling bias. If a player has a disaster year, a certain portion of that is bad luck. And a disproportionate number of “last year” performances has more bad luck than good luck. That’s why it looks like you have a big drop in the last year. But that’s only because he wasn’t asked to come back the next year. Well, for those who WERE asked to come back the next year following a disaster year, then, guess what, that disaster year no longer is part of the “last year”. Hence, the bias.
When constructing these aging curves, it is ALWAYS ALWAYS ALWAYS important to pretend that the future has not existed. If you want to know Peyton Manning’s aging pattern from 2011 onwards, then you have to look at other QB who managed to play at a high level at the age of 35. And do NOT look at how (or even if) they played from age 36 onwards in selecting your players. The future is unknown, from the perspective of SELECTING your players.
Once the selection process has taken place, you then do look to their “futures”. Basically, you time travel back to each QB’s point in time such that you are in the same boat as you are with Peyton. And then you travel forward in time for each QB. And since we can’t do that with Peyton without our favorite Delorean, we simply estimate that by looking at all his comparables.
Clay presents an update, including a nice graph (if you click on the division header):
Cool Standings also has a nice chart, but they do it only by team, not division (the red line is what you want):
According to J-doug, that’s the case as of today.
It also seems to me that that must have been pretty close to what the odds may have said at the start of the season, too, no? Maybe it was 40% back then? Someone know?
That is, we’ve played 125 games to get to the point of where we expected to be all along. Where’s the Law & Order twist that should happen right around now?
These are all the pitchers born since 1952 (i.e., after Blyleven), with at least 60 starts, through age 27. They are ordered by ERA+ from worst to… uh, not-worst. The general rule is that if you’ve been given 60 starts, you are not going to get many more. The more non-worst you are, the more chances you are given. That’s why you see near the non-worst point, guys with over 100 starts. Kyle Davies stands out as someone who has been given an enormous number of starts at such a poor performance level. You’ll also note that a good portion of the players were given a fair number of games in the bullpen. Basically: it’s not working out here, let’s try you over there. Kyle Davies stands out as someone who was kept in the starting rotation. At some point, scouting has to give in to empirical data: as much as a scout may say that Kyle Davies is a decent pitcher, we have to accept that maybe he’s not that good.
Of course, right behind Davies is Mike Scott. You kids may not remember him, but he was one of the best pitchers of his era (at some point in his career).
Let’s look at the top 10 in this list, and see what they did in the 4 years after their age 27 season. How much hope can we possibly give the Kyle Davies of the world? (I had to exclude pitchers who are still to young, so I went down to the #15 on the list below to get my top 10.)
Here we go:
Van Poppel: 282 IP, 108 ERA+ (almost all in relief)
Bowen: out of MLB
Mike Scott: 796 IP, 106 ERA+ (almost all as starter), and then continued on for a few more excellent seasons)
Knapp: out of MLB
Snyder: out of MLB
Scudder: out of MLB
Walk: 530 IP, 115 ERA+ (half games as starter), and continued his career beyond
Rupe: 10 more innings then out of MLB
Wright: out of MLB
Codiroli (*): 122 IP, 75 ERA+
(*) I followed MLB intently when I was a kid, knowing every player on every team (easier done when you collect baseball cards, and are in fantasy leagues). I do not remember this guy at all.
So, that’s what you have here: 20% chance of good success, 10% chance of being useful as back of the bullpen guy, and 70% chance of being out of MLB.
Note: replacement level is ERA+ of 75-80 as a starter, and 95 as a reliever.
Glove-slap: Eric.
Source Baseball-Reference.com:
I like the idea, and the anectodal evidence. Now, you just need to quantify it in a more systematic fashion.
May 16 22:50
Dodgers’ win reversed because Mattingly did not attest to proper score!
May 16 20:44
How to beat the shift
May 16 20:02
Sponsoring MLB jerseys
May 16 19:34
Now you frame it, now you don’t
May 16 16:56
Did Manny Pacquaio actually quote Leviticus?
May 16 16:06
Does changing your pitch frequency lead to substantial change in results?
May 16 14:18
Extra Innings: One-minute review
May 16 14:16
This particular criticism of UZR is unfounded
May 16 13:21
Psst… wanna intern for the Astros?
May 16 12:23
Arena wars
THREADS
May 16, 2012
Now you frame it, now you don’t
May 16, 2012
Dodgers’ win reversed because Mattingly did not attest to proper score!
May 16, 2012
Does changing your pitch frequency lead to substantial change in results?
May 16, 2012
Sponsoring MLB jerseys
May 15, 2012
Andre The Hawk Dawson speaks
May 15, 2012
Euro 2012 Preview
May 15, 2012
How to beat the shift
May 15, 2012
Will Pujols end the season with at least 30 HR and .500 SLG?
May 15, 2012
Kershaw v Strasburg, part 2
May 15, 2012
Did Manny Pacquaio actually quote Leviticus?
Recent comments
Older comments
Page 3 of 342 pages « First < 1 2 3 4 5 > Last »Complete Archive – By Category
Complete Archive – By Date