THE BOOK--Playing The Percentages In Baseball

http://www.bbnflstats.com/2007/09/pre-season-predictions-are-worthless.html

Over the past two years, the Football Outsiders predictions are no better than the consensus Vegas lines. They are wrong by an average 2.6 wins for each team. Although the NFL is difficult to predict, 2.6 games is quite a bit in a 16 game season.

How do we judge whether 2.6 games is a good or bad prediction? To judge if predictions are worth anything, we should compare them to obvious knowledge. I compared the FO and Vegas predictions to two sets of obvious predictions. The first is if I mindlessly predicted 8 wins for every team. The second is just using a regression of last year’s wins… The average errors for ‘05 and ‘06 was 2.5 games for both the mindless 8-game predictions and last year’s records. Both “obvious” methods actually do slightly better than the expert predictions.

This is the same kind of result we get with forecasting players.  Anyone who touts their forecasting system as anything better than a smidge better than the market is really doing everyone a disservice.  Which is why it bothered me to read this from Bill James:

First, I knew that our hitting projections for minor leaguers have a very high degree of reliability—much higher than most people assume that they do.

I would assume that forecasting 22 year old minor leaguers (with 300-500 PA at age 22) playing MLB at age 23 to be somewhat worse than forecasting 22 year old MLB players (with 300-500 PA at age 22) playing MLB at age 23.

How much worse?  I don’t know.  But, I’d guess about on par with forecasting 22 year old MLB players playing MLB at age 24.

“High-degree of reliability”?  That’s as misleading as saying “deadly accurate”. 

Shame on trumpeting forecasters who claim any accuracy beyond a smidge.

Fun article. I once tried to create a wiki page over on SoSH to dump writers’ & posters’ predictions for Red Sox prospects as they came in, but then I couldn’t figure out how to do it, and then I lost interest. Still think it would make for interesting reading.

On a somewhat related note, FutureScanner take a Digg-style approach to tracking predictions as they are made in news stories:

http://memebox.com/futurescanner

On asset prices, forecasters are very close to 100% useless.  The market price or futures price are the result of millions of people making predictions and putting their own money on them.

Asking 60 economists to predict the price of oil, out of thousands of people who also predict the price of oil *and put their own money on their predictions*, can’t be of much use.  You’re taking thousands of people, picking 60 of them, and assuming the 60 are more accurate than the thousands. 

Also, as I think you and MGL have said before: if you are able to put money on your predictions, and you choose not to, then you’re probably just a bullsh-tter.

Right, if you can’t take yourself seriously enough that you stand behind your words and are proactively accountable, how is someone else supposed to?

Just a bunch of yappers…

Disagreeing with Colbert???

I’m sure he could tie the yapping back to bears, though. (And I’ve been an Easterbrook fan for a while.)

While there’s no doubt that opinions backed by bets deserve more weight than those that are not, this financial snippet from Easterbrook’s column is misleading for two reasons.

One, these predictions are only partially independent. The fed funds rate was lower than expected because the Fed was more active about cutting rates than expected. This translates to a weaker dollar, all other things being equal. So the predictions about the euro, the price of oil (which is expressed in USD), and inflation are all less surprising to various degrees, given that the fed funds rate prediction was off. It’s inaccurate to characterize them as scattershot predictions, one of which will hopefully be right.

Two, the economists’ predictions are not very different from what the futures markets were predicting. Easterbrook makes them sound like they pulled the numbers out of nowhere. But at the start of 2007, Dec07 oil futures were right around $60. The 1 year euro forward was in the 1.31-1.33 range. Some of the economists probably pulled their predictions straight from the markets. And even if they didn’t, it would be very odd for the average predictions of a panel of 60 professional market watchers to be substantially different from implied market levels. We should expect their consensus to be close to the traded market’s consensus.

It’s important for someone to hold pundits accountable, but with his smarminess and self-satisfaction, Easterbrook goes one step further and becomes a pundit himself. If Easterbrook wants to bet that long-dated oil futures are wrong this year, there are people standing ready to accommodate his wishes.

I think that was his point.  Why bother asking “60 top people” about something, if the market is hard to beat?  Same as the Vegas line, the Marcels, the Community Forecasts, etc.  The crowd knows more than a handful of people.

Tango - Do you have any evidence that “the crowd knows more than a handful of people”?  It would seem to be dependent on which “handful of people” and which crowd.  The Marcels are not an example of this since the Marcels is basically the product of one very knowledgeble person without crowd input.  As far as I know, the Community Forecasts, although valuable information, has not been proven more accurate than a handful of knowlegeble analysts.  The Vegas line often changes significantly from when it is first posted the day before a game until it closes at gametime.  Some of those changes may be do to new information becoming available, but most of the change is a response to incoming bets from “the crowd”.  While I have thought that this might make an interesting study between the experts forecasting ability ("the early line") versus the crowds ("the closing line"), I am unaware that any study has already been done.