THE BOOK--Playing The Percentages In Baseball

Wednesday, December 01, 2010

Conclusions?

I was bothered by Kahrl’s position here:

The problem is that, substantively, where defense is concerned we’re really no better off than we were a decade ago. As entertaining as it might be that we’ve added a lot of additional tools to the defensive arsenal, they remain a crude lot, and we’re still left with broad-swipe metrics that haven’t really significantly advanced our understanding from where we were almost 20 years ago with Sherri Nichols’ Defensive Average or Clay Davenport’s Fielding Runs. In the ‘90s, we saw horribly flawed, yet inevitably interesting tools like Zone Rating and then Revised Zone Rating get touted because they numericized worthy ambitions, only to get quietly, deservingly chucked under the bus by many statheads (including Bill James, among others) more than a decade ago.

In the years since, variety, in the form of UZR and Total Zone and Plus/Minus, of Dan Fox’s SFR and Dave Pinto’s Probabilistic Model, and more besides, have not delivered conclusive answers. They are all interesting cracks at the problem, each entertaining and worthy of consideration as we seek to expand what we think we know and what we want to know. But because we’re still at the point where some researchers [editor’s note: she means MGL] say in so many words, “I don’t know if +10 is actually better or worse than -10” in describing one defensive stat’s output—something we need to say about all of them, albeit without chucking any of them—we’re still essentially left looking at the full spread of fielding statistics and hoping that the answers are consistent enough about any individual player across time and multiple metrics that it suggests something like the truth. Even if Derek Jeter is consistently lousy across every metric, year after year, it doesn’t prove he’s lousy, but it does constitute a fairly unsubtle hint. To get overly hung up on the numeric values themselves is to accord them greater accuracy than they deserve when it comes to picking between players.

I don’t think she made a good characterization here.  I think she had a definite bias against fielding metrics in general, and zone rating in particular, and picked and choose her arguments to support her position, while giving just the faintest support for them.

My reply:

Christina said:
“The problem is that, substantively, where defense is concerned we’re really no better off than we were a decade ago. “

That is a terrible conclusion to Colin’s article. Christina echoed a reader in the linked article, who said:
“Seeing as we haven’t made much progress in 20 years with defensive metrics...”

This is what I said in the comments section in reply to that statement:

============= Tangotiger ==============
However, I am bothered that a reader, after reading Colin’s piece, would come to a conclusion like:

“Seeing as we haven’t made much progress in 20 years with defensive metrics...”

The only fair conclusion to make is that we don’t know how much progress we have made, and not that we haven’t made much progress.

We’ve made “some”. Is that a little? A lot? You can’t say “not much”. This is part of the nuance in Colin’s piece that may be glossed over, if said reader is representative of a portion of the readership.
==========================================

This is what Mike Fast said in reply to that reader:

=============== Mike Fast =================
Colin said he wasn’t sure and wasn’t sure how to tell. You seemed to wipe away the uncertainty and conclude that we have made no progress.

There is reason to believe we have made progress, certainly on the theory side. But until we can test, we won’t know for sure. That’s historically been the standard in sabermetrics.

However, people who have developed fielding metrics will take your criticism very differently if you say, “I don’t see how to tell how accurate your metric is” versus “Your metric is worthless.” The former is a statement of fact that can be contested and explained, though it may raise some emotions. The second is a very value judgment that comes across as very dismissive and not focused on the examination of facts.
==========================================

As for the +10 and -10 argument, that applies to everything, be it fielding, offense, pitching, hockey, football, car accidents… these are all results of sample observations.